pyneoinstance 1.1.0__tar.gz → 1.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {pyneoinstance-1.1.0/pyneoinstance.egg-info → pyneoinstance-1.1.2}/PKG-INFO +70 -37
- pyneoinstance-1.1.2/README.md +236 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/__init__.py +1 -1
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/database/neo4jdbms.py +17 -14
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2/pyneoinstance.egg-info}/PKG-INFO +70 -37
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyproject.toml +2 -2
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/test/cypher_queries.yaml +9 -6
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/test/test_neo4jdb.py +12 -2
- pyneoinstance-1.1.0/README.md +0 -206
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/LICENSE +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/MANIFEST.in +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/database/__init__.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/database/context.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/fileload/__init__.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/fileload/context.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/fileload/helpers.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/fileload/loadyaml.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/util/__init__.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance/util/functions.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance.egg-info/SOURCES.txt +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance.egg-info/dependency_links.txt +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance.egg-info/requires.txt +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/pyneoinstance.egg-info/top_level.txt +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/setup.cfg +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/setup.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/test/load_files.yaml +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/test/movies.csv +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/test/people.csv +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/test/roles.csv +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/test/test.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/test/test_fileload.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-1.1.2}/test/test_util_functions.py +0 -0
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: pyneoinstance
|
|
3
|
-
Version: 1.1.
|
|
3
|
+
Version: 1.1.2
|
|
4
4
|
Summary: Handle Neo4j read and write transactions
|
|
5
|
-
Author-email: Jose A Alvarado-
|
|
5
|
+
Author-email: "Jose A. Alvarado-Guzman" <jose.alvarado-guzman@neo4j.com>
|
|
6
6
|
License: GNU GENERAL PUBLIC LICENSE
|
|
7
7
|
Version 3, 29 June 2007
|
|
8
8
|
|
|
@@ -685,17 +685,22 @@ Classifier: Development Status :: 4 - Beta
|
|
|
685
685
|
Requires-Python: >=3.8
|
|
686
686
|
Description-Content-Type: text/markdown
|
|
687
687
|
License-File: LICENSE
|
|
688
|
+
Requires-Dist: neo4j>=5.0.0
|
|
689
|
+
Requires-Dist: pandas
|
|
690
|
+
Requires-Dist: PyYAML
|
|
688
691
|
|
|
689
692
|
# PyNeoInstance Package
|
|
690
693
|
|
|
691
694
|
## What is it?
|
|
692
695
|
|
|
693
|
-
|
|
696
|
+
PyNeoInstance is a Python package that provides a user friendly API for submitting Cypher queries to Neo4j.
|
|
694
697
|
It supports the following features:
|
|
695
698
|
|
|
696
|
-
- __Read query__: Submit a Cypher query and get
|
|
697
|
-
- __Write query__:
|
|
698
|
-
- __Write
|
|
699
|
+
- __Read query__: Submit a Cypher read query, with or without parameters, and get a Pandas Data Frame with the results.
|
|
700
|
+
- __Write query__: Submit a write Cypher query, with or without parameters, to update an existing database.
|
|
701
|
+
- __Write queries__: Submit a list of Cypher queries to update an existing database.
|
|
702
|
+
- __Write query with data__: Update a database based on a Pandas Data Frame and a Cypher query.
|
|
703
|
+
- __Write queries with data__: Update a database based on a Pandas Data Frame and a list of Cypher queries.
|
|
699
704
|
|
|
700
705
|
## How to install it?
|
|
701
706
|
|
|
@@ -707,16 +712,16 @@ pip install pyneoinstance
|
|
|
707
712
|
|
|
708
713
|
### Instantiating the Neo4jInstance class
|
|
709
714
|
|
|
710
|
-
In order to use this package you first need to import the Neo4jInstance class.
|
|
715
|
+
In order to use this package, you first need to import the Neo4jInstance class.
|
|
711
716
|
|
|
712
717
|
```python
|
|
713
|
-
from
|
|
718
|
+
from pyneoinstance import Neo4jInstance
|
|
714
719
|
```
|
|
715
720
|
|
|
716
721
|
Once imported, you need to instantiate this class by providing the following arguments:
|
|
717
722
|
|
|
718
723
|
- _uri_: The Neo4j instance unique resource identifier.
|
|
719
|
-
- _user_: The user name
|
|
724
|
+
- _user_: The user name used to authenticate.
|
|
720
725
|
- _password_: The user name password.
|
|
721
726
|
|
|
722
727
|
```python
|
|
@@ -737,7 +742,7 @@ user = os.environ['NEO4J_USER']
|
|
|
737
742
|
password = os.environ['NEO4J_PASSWORD']
|
|
738
743
|
graph = Neo4jInstance(uri,user,password)
|
|
739
744
|
```
|
|
740
|
-
Another alternative is to create a YAML
|
|
745
|
+
Another alternative is to create a YAML configuration file to store this information.
|
|
741
746
|
|
|
742
747
|
```yaml
|
|
743
748
|
# configuration.yaml
|
|
@@ -745,9 +750,10 @@ db_info:
|
|
|
745
750
|
uri: neo4j_uri
|
|
746
751
|
user: neo4j_user
|
|
747
752
|
password: user_password
|
|
753
|
+
database: neo4j
|
|
748
754
|
```
|
|
749
755
|
|
|
750
|
-
This YAML file can then be loaded in Python and
|
|
756
|
+
This YAML file can then be loaded in Python and this information can be retrieve.
|
|
751
757
|
|
|
752
758
|
```python
|
|
753
759
|
from pyneoinstance import load_yaml_file
|
|
@@ -758,7 +764,7 @@ graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
|
758
764
|
```
|
|
759
765
|
### Submitting a Cypher read query
|
|
760
766
|
|
|
761
|
-
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method
|
|
767
|
+
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receives the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query are return as a Pandas Data Frame. To avoid unwanted column names in the resulting Data Frame, always provide aliases for all returned properties.
|
|
762
768
|
|
|
763
769
|
```python
|
|
764
770
|
query = """
|
|
@@ -768,35 +774,56 @@ query = """
|
|
|
768
774
|
m.tagline AS tagline;
|
|
769
775
|
"""
|
|
770
776
|
|
|
771
|
-
data = graph.execute_read_query(query)
|
|
777
|
+
data = graph.execute_read_query(query,db_info['database'])
|
|
772
778
|
```
|
|
779
|
+
|
|
780
|
+
### Submitting a Cypher read query with parameters
|
|
781
|
+
|
|
782
|
+
The method execute\_read\_query can optionally received a Python Dictionary with the parameters required as part of you Cypher query.
|
|
783
|
+
|
|
784
|
+
```python
|
|
785
|
+
query = """
|
|
786
|
+
MATCH (m:Movie {title: $movie_title})
|
|
787
|
+
RETURN m.tile AS title,
|
|
788
|
+
m.released AS releasedYear,
|
|
789
|
+
m.tagline AS tagline;
|
|
790
|
+
"""
|
|
791
|
+
|
|
792
|
+
data = graph.execute_read_query(query, db_info['database'], db_info['database'], parameters={'movie_title':'The Matrix'})
|
|
793
|
+
```
|
|
794
|
+
|
|
773
795
|
### Submitting a Cypher write query
|
|
774
796
|
|
|
775
|
-
|
|
797
|
+
You can submit a Cypher write query using the method execute_write_query. The signature of this method is identical to the execute_read_query. This method returns a dictionary containing the updates made to the database. As with read queries you can also specify a Python dictionary containing Cypher parameters.
|
|
776
798
|
|
|
777
799
|
```python
|
|
778
800
|
query = """
|
|
779
|
-
MERGE(m:Movie {title
|
|
801
|
+
MERGE(m:Movie {title:$movie_title})
|
|
780
802
|
ON CREATE SET
|
|
781
|
-
m.release=
|
|
782
|
-
m.tagline =
|
|
803
|
+
m.release = $year,
|
|
804
|
+
m.tagline = $tagline
|
|
783
805
|
"""
|
|
784
806
|
|
|
785
|
-
results = graph.execute_write_query(query
|
|
807
|
+
results = graph.execute_write_query(query, db_info['database'],
|
|
808
|
+
parameters={
|
|
809
|
+
'movie_title':'Wakanda Forever',
|
|
810
|
+
'year'=2022,
|
|
811
|
+
tagline=''Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!''
|
|
812
|
+
})
|
|
786
813
|
```
|
|
787
814
|
|
|
788
815
|
### Updating a database with data in a DataFrame: Sequentially
|
|
789
816
|
|
|
790
|
-
We can update a Neo4j database with data contained in a Pandas
|
|
817
|
+
We can update a Neo4j database with data contained in a Pandas Data Frame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methods return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
791
818
|
|
|
792
|
-
- _data_: Pandas
|
|
793
|
-
- _database_: Optional, database on which to execute the
|
|
794
|
-
- _partitions_: Optional, number of partitions use to break the
|
|
795
|
-
- _parallel_:
|
|
819
|
+
- _data_: Pandas Data Frame containing data to process.
|
|
820
|
+
- _database_: Optional, database on which to execute the queries.
|
|
821
|
+
- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
|
|
822
|
+
- _parallel_: Whether to execute the query using multiple Python process.
|
|
796
823
|
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
797
824
|
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
798
825
|
|
|
799
|
-
To illustrates how to use
|
|
826
|
+
To illustrates how to use these methods lets imagine we have a csv file that looks like this:
|
|
800
827
|
|
|
801
828
|
```python
|
|
802
829
|
# people.csv
|
|
@@ -812,11 +839,11 @@ personId,name,birthYear,deathYear
|
|
|
812
839
|
3482,Chus Lampreave,1930,
|
|
813
840
|
```
|
|
814
841
|
|
|
815
|
-
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to
|
|
842
|
+
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performed the following steps:
|
|
816
843
|
|
|
817
|
-
1. Load the csv to Python as a
|
|
818
|
-
2. Develop the Cypher query that is going to be use to load the
|
|
819
|
-
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the
|
|
844
|
+
1. Load the csv to Python as a Data Frame.
|
|
845
|
+
2. Develop the Cypher query that is going to be use to load the Data Frame into Neo4j.
|
|
846
|
+
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the Data Frame, Cypher query and any other optional arguments to this method.
|
|
820
847
|
|
|
821
848
|
```python
|
|
822
849
|
import pandas as pd
|
|
@@ -834,10 +861,10 @@ person_load_query = """
|
|
|
834
861
|
p.deathYear = toInteger(row.deathYear)
|
|
835
862
|
"""
|
|
836
863
|
|
|
837
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
864
|
+
person_load_result = graph.execute_write_query_with_data(person_load_query, person_data, db_info['database'])
|
|
838
865
|
```
|
|
839
866
|
|
|
840
|
-
|
|
867
|
+
It is not a good practice to store your Cypher queries as strings within your Python modules. To solve this, we can extend our YAML configuration file to store all our Cypher queries.
|
|
841
868
|
|
|
842
869
|
```yaml
|
|
843
870
|
# configuration.yaml
|
|
@@ -845,9 +872,10 @@ db_info:
|
|
|
845
872
|
uri: neo4j_uri
|
|
846
873
|
user: neo4j_user
|
|
847
874
|
password: user_password
|
|
875
|
+
database: neo4j
|
|
848
876
|
loading_queries:
|
|
849
877
|
nodes:
|
|
850
|
-
|
|
878
|
+
Person: |
|
|
851
879
|
WITH $rows AS rows
|
|
852
880
|
UNWIND rows AS row
|
|
853
881
|
MERGE(p:Person {id:row.personId})
|
|
@@ -867,28 +895,33 @@ person_data = pd.read_csv('people.csv')
|
|
|
867
895
|
|
|
868
896
|
node_load_queries = config['loading_queries']['nodes']
|
|
869
897
|
|
|
870
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
898
|
+
person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
|
|
871
899
|
```
|
|
872
900
|
|
|
873
|
-
If the number of rows in your
|
|
901
|
+
If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
874
902
|
|
|
875
903
|
```python
|
|
876
904
|
person_load_result = graph.execute_write_query_with_data(
|
|
877
|
-
person_data,
|
|
878
905
|
node_load_queries['person'],
|
|
906
|
+
person_data,
|
|
907
|
+
db_info['database'],
|
|
879
908
|
partitions = 5
|
|
880
909
|
)
|
|
881
910
|
```
|
|
882
|
-
## Updating a database with data in a
|
|
911
|
+
## Updating a database with data in a Data Frame: Parallel
|
|
883
912
|
|
|
884
|
-
To load the data in the
|
|
913
|
+
To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
|
|
885
914
|
|
|
886
915
|
```python
|
|
887
916
|
person_load_result = graph.execute_write_query_with_data(
|
|
888
|
-
person_data,
|
|
889
917
|
node_load_queries['person'],
|
|
918
|
+
person_data,
|
|
890
919
|
partitions = 5,
|
|
891
920
|
parallel = True,
|
|
892
921
|
workers = 5
|
|
893
922
|
)
|
|
894
923
|
```
|
|
924
|
+
|
|
925
|
+
## More detailed examples
|
|
926
|
+
|
|
927
|
+
For more detailed examples please referred to this [article](https://medium.com/@a-gilmore/pyneoinstance-a-user-friendly-python-library-for-neo4j-dbefa3117bb2).
|
|
@@ -0,0 +1,236 @@
|
|
|
1
|
+
# PyNeoInstance Package
|
|
2
|
+
|
|
3
|
+
## What is it?
|
|
4
|
+
|
|
5
|
+
PyNeoInstance is a Python package that provides a user friendly API for submitting Cypher queries to Neo4j.
|
|
6
|
+
It supports the following features:
|
|
7
|
+
|
|
8
|
+
- __Read query__: Submit a Cypher read query, with or without parameters, and get a Pandas Data Frame with the results.
|
|
9
|
+
- __Write query__: Submit a write Cypher query, with or without parameters, to update an existing database.
|
|
10
|
+
- __Write queries__: Submit a list of Cypher queries to update an existing database.
|
|
11
|
+
- __Write query with data__: Update a database based on a Pandas Data Frame and a Cypher query.
|
|
12
|
+
- __Write queries with data__: Update a database based on a Pandas Data Frame and a list of Cypher queries.
|
|
13
|
+
|
|
14
|
+
## How to install it?
|
|
15
|
+
|
|
16
|
+
```python
|
|
17
|
+
pip install pyneoinstance
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## How to use it?
|
|
21
|
+
|
|
22
|
+
### Instantiating the Neo4jInstance class
|
|
23
|
+
|
|
24
|
+
In order to use this package, you first need to import the Neo4jInstance class.
|
|
25
|
+
|
|
26
|
+
```python
|
|
27
|
+
from pyneoinstance import Neo4jInstance
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Once imported, you need to instantiate this class by providing the following arguments:
|
|
31
|
+
|
|
32
|
+
- _uri_: The Neo4j instance unique resource identifier.
|
|
33
|
+
- _user_: The user name used to authenticate.
|
|
34
|
+
- _password_: The user name password.
|
|
35
|
+
|
|
36
|
+
```python
|
|
37
|
+
uri = 'bolt://localhost:7687'
|
|
38
|
+
user = 'neo4j'
|
|
39
|
+
password = 'password'
|
|
40
|
+
|
|
41
|
+
graph = Neo4jInstance(uri, user, password)
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Although the previous code is valid, it is not recommended to hard code your user credentials in your application. As an alternative you can store this information as part of your OS user environment variables and then retrieve them in Python.
|
|
45
|
+
|
|
46
|
+
```python
|
|
47
|
+
import os
|
|
48
|
+
|
|
49
|
+
uri = os.environ['NEO4J_URI']
|
|
50
|
+
user = os.environ['NEO4J_USER']
|
|
51
|
+
password = os.environ['NEO4J_PASSWORD']
|
|
52
|
+
graph = Neo4jInstance(uri,user,password)
|
|
53
|
+
```
|
|
54
|
+
Another alternative is to create a YAML configuration file to store this information.
|
|
55
|
+
|
|
56
|
+
```yaml
|
|
57
|
+
# configuration.yaml
|
|
58
|
+
db_info:
|
|
59
|
+
uri: neo4j_uri
|
|
60
|
+
user: neo4j_user
|
|
61
|
+
password: user_password
|
|
62
|
+
database: neo4j
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
This YAML file can then be loaded in Python and this information can be retrieve.
|
|
66
|
+
|
|
67
|
+
```python
|
|
68
|
+
from pyneoinstance import load_yaml_file
|
|
69
|
+
|
|
70
|
+
config = load_yaml_file(configuration.yaml)
|
|
71
|
+
db_info = config['db_info']
|
|
72
|
+
graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
73
|
+
```
|
|
74
|
+
### Submitting a Cypher read query
|
|
75
|
+
|
|
76
|
+
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receives the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query are return as a Pandas Data Frame. To avoid unwanted column names in the resulting Data Frame, always provide aliases for all returned properties.
|
|
77
|
+
|
|
78
|
+
```python
|
|
79
|
+
query = """
|
|
80
|
+
MATCH (m:Movie {title: "The Matrix"})
|
|
81
|
+
RETURN m.tile AS title,
|
|
82
|
+
m.released AS releasedYear,
|
|
83
|
+
m.tagline AS tagline;
|
|
84
|
+
"""
|
|
85
|
+
|
|
86
|
+
data = graph.execute_read_query(query,db_info['database'])
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
### Submitting a Cypher read query with parameters
|
|
90
|
+
|
|
91
|
+
The method execute\_read\_query can optionally received a Python Dictionary with the parameters required as part of you Cypher query.
|
|
92
|
+
|
|
93
|
+
```python
|
|
94
|
+
query = """
|
|
95
|
+
MATCH (m:Movie {title: $movie_title})
|
|
96
|
+
RETURN m.tile AS title,
|
|
97
|
+
m.released AS releasedYear,
|
|
98
|
+
m.tagline AS tagline;
|
|
99
|
+
"""
|
|
100
|
+
|
|
101
|
+
data = graph.execute_read_query(query, db_info['database'], db_info['database'], parameters={'movie_title':'The Matrix'})
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### Submitting a Cypher write query
|
|
105
|
+
|
|
106
|
+
You can submit a Cypher write query using the method execute_write_query. The signature of this method is identical to the execute_read_query. This method returns a dictionary containing the updates made to the database. As with read queries you can also specify a Python dictionary containing Cypher parameters.
|
|
107
|
+
|
|
108
|
+
```python
|
|
109
|
+
query = """
|
|
110
|
+
MERGE(m:Movie {title:$movie_title})
|
|
111
|
+
ON CREATE SET
|
|
112
|
+
m.release = $year,
|
|
113
|
+
m.tagline = $tagline
|
|
114
|
+
"""
|
|
115
|
+
|
|
116
|
+
results = graph.execute_write_query(query, db_info['database'],
|
|
117
|
+
parameters={
|
|
118
|
+
'movie_title':'Wakanda Forever',
|
|
119
|
+
'year'=2022,
|
|
120
|
+
tagline=''Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!''
|
|
121
|
+
})
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### Updating a database with data in a DataFrame: Sequentially
|
|
125
|
+
|
|
126
|
+
We can update a Neo4j database with data contained in a Pandas Data Frame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methods return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
127
|
+
|
|
128
|
+
- _data_: Pandas Data Frame containing data to process.
|
|
129
|
+
- _database_: Optional, database on which to execute the queries.
|
|
130
|
+
- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
|
|
131
|
+
- _parallel_: Whether to execute the query using multiple Python process.
|
|
132
|
+
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
133
|
+
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
134
|
+
|
|
135
|
+
To illustrates how to use these methods lets imagine we have a csv file that looks like this:
|
|
136
|
+
|
|
137
|
+
```python
|
|
138
|
+
# people.csv
|
|
139
|
+
personId,name,birthYear,deathYear
|
|
140
|
+
23945,Gérard Pirès,1942,
|
|
141
|
+
553509,Helen Reddy,1941,
|
|
142
|
+
113934,Susan Flannery,1939,
|
|
143
|
+
26706,David Rintoul,1948,
|
|
144
|
+
237537,Rita Marley,1946,
|
|
145
|
+
11502,Harry Davenport,1866,1949
|
|
146
|
+
11792,George Dickerson,1933,
|
|
147
|
+
7242,Temuera Morrison,1960,
|
|
148
|
+
3482,Chus Lampreave,1930,
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performed the following steps:
|
|
152
|
+
|
|
153
|
+
1. Load the csv to Python as a Data Frame.
|
|
154
|
+
2. Develop the Cypher query that is going to be use to load the Data Frame into Neo4j.
|
|
155
|
+
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the Data Frame, Cypher query and any other optional arguments to this method.
|
|
156
|
+
|
|
157
|
+
```python
|
|
158
|
+
import pandas as pd
|
|
159
|
+
|
|
160
|
+
person_data = pd.read_csv('people.csv')
|
|
161
|
+
|
|
162
|
+
person_load_query = """
|
|
163
|
+
WITH $rows AS rows
|
|
164
|
+
UNWIND rows AS row
|
|
165
|
+
MERGE(p:Person {id:row.personId})
|
|
166
|
+
ON CREATE
|
|
167
|
+
SET p.id = row.personId,
|
|
168
|
+
p.name = row.name,
|
|
169
|
+
p.birthYear=toInteger(row.birthYear),
|
|
170
|
+
p.deathYear = toInteger(row.deathYear)
|
|
171
|
+
"""
|
|
172
|
+
|
|
173
|
+
person_load_result = graph.execute_write_query_with_data(person_load_query, person_data, db_info['database'])
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
It is not a good practice to store your Cypher queries as strings within your Python modules. To solve this, we can extend our YAML configuration file to store all our Cypher queries.
|
|
177
|
+
|
|
178
|
+
```yaml
|
|
179
|
+
# configuration.yaml
|
|
180
|
+
db_info:
|
|
181
|
+
uri: neo4j_uri
|
|
182
|
+
user: neo4j_user
|
|
183
|
+
password: user_password
|
|
184
|
+
database: neo4j
|
|
185
|
+
loading_queries:
|
|
186
|
+
nodes:
|
|
187
|
+
Person: |
|
|
188
|
+
WITH $rows AS rows
|
|
189
|
+
UNWIND rows AS row
|
|
190
|
+
MERGE(p:Person {id:row.personId})
|
|
191
|
+
ON CREATE
|
|
192
|
+
SET p.id = row.personId,
|
|
193
|
+
p.name = row.name,
|
|
194
|
+
p.birthYear=toInteger(row.birthYear),
|
|
195
|
+
p.deathYear = toInteger(row.deathYear)
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Now that we added the loading query to our configuration file, lets re-write our Python script to get this query from the configuration file.
|
|
199
|
+
|
|
200
|
+
```python
|
|
201
|
+
import pandas as pd
|
|
202
|
+
|
|
203
|
+
person_data = pd.read_csv('people.csv')
|
|
204
|
+
|
|
205
|
+
node_load_queries = config['loading_queries']['nodes']
|
|
206
|
+
|
|
207
|
+
person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
211
|
+
|
|
212
|
+
```python
|
|
213
|
+
person_load_result = graph.execute_write_query_with_data(
|
|
214
|
+
node_load_queries['person'],
|
|
215
|
+
person_data,
|
|
216
|
+
db_info['database'],
|
|
217
|
+
partitions = 5
|
|
218
|
+
)
|
|
219
|
+
```
|
|
220
|
+
## Updating a database with data in a Data Frame: Parallel
|
|
221
|
+
|
|
222
|
+
To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
|
|
223
|
+
|
|
224
|
+
```python
|
|
225
|
+
person_load_result = graph.execute_write_query_with_data(
|
|
226
|
+
node_load_queries['person'],
|
|
227
|
+
person_data,
|
|
228
|
+
partitions = 5,
|
|
229
|
+
parallel = True,
|
|
230
|
+
workers = 5
|
|
231
|
+
)
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
## More detailed examples
|
|
235
|
+
|
|
236
|
+
For more detailed examples please referred to this [article](https://medium.com/@a-gilmore/pyneoinstance-a-user-friendly-python-library-for-neo4j-dbefa3117bb2).
|
|
@@ -20,10 +20,8 @@ from neo4j.exceptions import ServiceUnavailable
|
|
|
20
20
|
from neo4j.exceptions import AuthError
|
|
21
21
|
from neo4j.exceptions import ClientError
|
|
22
22
|
from neo4j.exceptions import ConfigurationError
|
|
23
|
-
from .context import get_logger
|
|
24
23
|
|
|
25
24
|
def _get_driver(neo_info : Dict[str, str]):
|
|
26
|
-
logger = get_logger('pyneoinstance', logging.INFO)
|
|
27
25
|
try:
|
|
28
26
|
if neo_info['encrypted'] != '':
|
|
29
27
|
driver = GraphDatabase.driver(
|
|
@@ -169,12 +167,11 @@ class Neo4jInstance:
|
|
|
169
167
|
self.neo_info['password'] = password
|
|
170
168
|
self.neo_info['encrypted'] = kwargs.get('encrypted') or ''
|
|
171
169
|
self.__results = defaultdict(int)
|
|
172
|
-
self.__logger = get_logger('pyneoinstance', logging.INFO)
|
|
173
170
|
|
|
174
171
|
|
|
175
172
|
def execute_read_query(self, query: str,
|
|
176
173
|
database: Optional[str] = None,
|
|
177
|
-
|
|
174
|
+
parameters: Optional[Dict[str, Any]] = None
|
|
178
175
|
) -> DataFrame:
|
|
179
176
|
"""Execute a read transaction to a specific database.
|
|
180
177
|
|
|
@@ -186,7 +183,7 @@ class Neo4jInstance:
|
|
|
186
183
|
Name of the Neo4j database of which to execute the
|
|
187
184
|
transaction. If not provided the default database is going to
|
|
188
185
|
be use.
|
|
189
|
-
|
|
186
|
+
parameters: Dict[str, Any], optional
|
|
190
187
|
Extra arguments containing optional cypher parameters.
|
|
191
188
|
|
|
192
189
|
Return
|
|
@@ -201,11 +198,15 @@ class Neo4jInstance:
|
|
|
201
198
|
ClientError
|
|
202
199
|
When there is a Cypher syntax error or datatype error.
|
|
203
200
|
"""
|
|
201
|
+
if parameters:
|
|
202
|
+
params = parameters
|
|
203
|
+
else:
|
|
204
|
+
params = {}
|
|
204
205
|
with _get_driver(self.neo_info) as driver:
|
|
205
206
|
with _get_session(driver, database) as session:
|
|
206
207
|
try:
|
|
207
208
|
result = session.execute_read(
|
|
208
|
-
_read_transaction_function,query=query,**
|
|
209
|
+
_read_transaction_function,query=query,**params)
|
|
209
210
|
except ServiceUnavailable as exception:
|
|
210
211
|
raise ServiceUnavailable() from exception
|
|
211
212
|
except ClientError as exception:
|
|
@@ -214,7 +215,7 @@ class Neo4jInstance:
|
|
|
214
215
|
|
|
215
216
|
def execute_write_queries(self, queries: List[str],
|
|
216
217
|
database: Optional[str] = None,
|
|
217
|
-
|
|
218
|
+
parameters: Optional[Dict[str, Any]] = None
|
|
218
219
|
) -> Dict[str, int]:
|
|
219
220
|
"""Execute a write query to a specific database.
|
|
220
221
|
|
|
@@ -225,7 +226,7 @@ class Neo4jInstance:
|
|
|
225
226
|
database : str, optional
|
|
226
227
|
Name of the Neo4j database of which to execute the transaction.
|
|
227
228
|
If not provided the default database is going to be use.
|
|
228
|
-
|
|
229
|
+
parameters : Dict[str, Any], optional
|
|
229
230
|
Extra arguments containing optional cypher parameters.
|
|
230
231
|
|
|
231
232
|
Returns
|
|
@@ -241,20 +242,23 @@ class Neo4jInstance:
|
|
|
241
242
|
ClientError
|
|
242
243
|
When their is a Cypher syntax error or datatype error.
|
|
243
244
|
"""
|
|
245
|
+
if parameters:
|
|
246
|
+
params = parameters
|
|
247
|
+
else:
|
|
248
|
+
params = {}
|
|
244
249
|
results = defaultdict(int)
|
|
245
250
|
with _get_driver(self.neo_info) as driver:
|
|
246
251
|
with _get_session(driver, database) as session:
|
|
247
252
|
for query in queries:
|
|
248
|
-
result = _execute_write(session, query,
|
|
253
|
+
result = _execute_write(session, query, parameters=params)
|
|
249
254
|
for key, value in result.items():
|
|
250
255
|
if key != '_contains_updates':
|
|
251
256
|
results[key] += value
|
|
252
|
-
self.__logger.info(f'Loading stats: {dict(results)}')
|
|
253
257
|
return dict(results)
|
|
254
258
|
|
|
255
259
|
def execute_write_query(self, query: str,
|
|
256
260
|
database: Optional[str] = None,
|
|
257
|
-
|
|
261
|
+
parameters: Optional[Dict['str',Any]] = None
|
|
258
262
|
) -> Dict[str, Any]:
|
|
259
263
|
"""Execute a write query to a specific database.
|
|
260
264
|
|
|
@@ -265,7 +269,7 @@ class Neo4jInstance:
|
|
|
265
269
|
database : str, optional
|
|
266
270
|
Name of the Neo4j database of which to execute the transaction.
|
|
267
271
|
If not provided the default database is going to be use.
|
|
268
|
-
|
|
272
|
+
parameters : Dict[str, Any], optional
|
|
269
273
|
Extra arguments containing optional cypher parameters.
|
|
270
274
|
|
|
271
275
|
Returns
|
|
@@ -281,7 +285,7 @@ class Neo4jInstance:
|
|
|
281
285
|
ClientError
|
|
282
286
|
When their is a Cypher syntax error or datatype error.
|
|
283
287
|
"""
|
|
284
|
-
result = self.execute_write_queries([query], database,
|
|
288
|
+
result = self.execute_write_queries([query], database, parameters)
|
|
285
289
|
return result
|
|
286
290
|
|
|
287
291
|
def execute_write_queries_with_data(self, queries: List[str], data: DataFrame,
|
|
@@ -364,7 +368,6 @@ class Neo4jInstance:
|
|
|
364
368
|
self.__results[key] += value
|
|
365
369
|
results = dict(self.__results.copy())
|
|
366
370
|
self.__results.clear()
|
|
367
|
-
self.__logger.info(f'Loading stats: {results}')
|
|
368
371
|
return results
|
|
369
372
|
|
|
370
373
|
def execute_write_query_with_data(self,
|
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: pyneoinstance
|
|
3
|
-
Version: 1.1.
|
|
3
|
+
Version: 1.1.2
|
|
4
4
|
Summary: Handle Neo4j read and write transactions
|
|
5
|
-
Author-email: Jose A Alvarado-
|
|
5
|
+
Author-email: "Jose A. Alvarado-Guzman" <jose.alvarado-guzman@neo4j.com>
|
|
6
6
|
License: GNU GENERAL PUBLIC LICENSE
|
|
7
7
|
Version 3, 29 June 2007
|
|
8
8
|
|
|
@@ -685,17 +685,22 @@ Classifier: Development Status :: 4 - Beta
|
|
|
685
685
|
Requires-Python: >=3.8
|
|
686
686
|
Description-Content-Type: text/markdown
|
|
687
687
|
License-File: LICENSE
|
|
688
|
+
Requires-Dist: neo4j>=5.0.0
|
|
689
|
+
Requires-Dist: pandas
|
|
690
|
+
Requires-Dist: PyYAML
|
|
688
691
|
|
|
689
692
|
# PyNeoInstance Package
|
|
690
693
|
|
|
691
694
|
## What is it?
|
|
692
695
|
|
|
693
|
-
|
|
696
|
+
PyNeoInstance is a Python package that provides a user friendly API for submitting Cypher queries to Neo4j.
|
|
694
697
|
It supports the following features:
|
|
695
698
|
|
|
696
|
-
- __Read query__: Submit a Cypher query and get
|
|
697
|
-
- __Write query__:
|
|
698
|
-
- __Write
|
|
699
|
+
- __Read query__: Submit a Cypher read query, with or without parameters, and get a Pandas Data Frame with the results.
|
|
700
|
+
- __Write query__: Submit a write Cypher query, with or without parameters, to update an existing database.
|
|
701
|
+
- __Write queries__: Submit a list of Cypher queries to update an existing database.
|
|
702
|
+
- __Write query with data__: Update a database based on a Pandas Data Frame and a Cypher query.
|
|
703
|
+
- __Write queries with data__: Update a database based on a Pandas Data Frame and a list of Cypher queries.
|
|
699
704
|
|
|
700
705
|
## How to install it?
|
|
701
706
|
|
|
@@ -707,16 +712,16 @@ pip install pyneoinstance
|
|
|
707
712
|
|
|
708
713
|
### Instantiating the Neo4jInstance class
|
|
709
714
|
|
|
710
|
-
In order to use this package you first need to import the Neo4jInstance class.
|
|
715
|
+
In order to use this package, you first need to import the Neo4jInstance class.
|
|
711
716
|
|
|
712
717
|
```python
|
|
713
|
-
from
|
|
718
|
+
from pyneoinstance import Neo4jInstance
|
|
714
719
|
```
|
|
715
720
|
|
|
716
721
|
Once imported, you need to instantiate this class by providing the following arguments:
|
|
717
722
|
|
|
718
723
|
- _uri_: The Neo4j instance unique resource identifier.
|
|
719
|
-
- _user_: The user name
|
|
724
|
+
- _user_: The user name used to authenticate.
|
|
720
725
|
- _password_: The user name password.
|
|
721
726
|
|
|
722
727
|
```python
|
|
@@ -737,7 +742,7 @@ user = os.environ['NEO4J_USER']
|
|
|
737
742
|
password = os.environ['NEO4J_PASSWORD']
|
|
738
743
|
graph = Neo4jInstance(uri,user,password)
|
|
739
744
|
```
|
|
740
|
-
Another alternative is to create a YAML
|
|
745
|
+
Another alternative is to create a YAML configuration file to store this information.
|
|
741
746
|
|
|
742
747
|
```yaml
|
|
743
748
|
# configuration.yaml
|
|
@@ -745,9 +750,10 @@ db_info:
|
|
|
745
750
|
uri: neo4j_uri
|
|
746
751
|
user: neo4j_user
|
|
747
752
|
password: user_password
|
|
753
|
+
database: neo4j
|
|
748
754
|
```
|
|
749
755
|
|
|
750
|
-
This YAML file can then be loaded in Python and
|
|
756
|
+
This YAML file can then be loaded in Python and this information can be retrieve.
|
|
751
757
|
|
|
752
758
|
```python
|
|
753
759
|
from pyneoinstance import load_yaml_file
|
|
@@ -758,7 +764,7 @@ graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
|
758
764
|
```
|
|
759
765
|
### Submitting a Cypher read query
|
|
760
766
|
|
|
761
|
-
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method
|
|
767
|
+
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receives the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query are return as a Pandas Data Frame. To avoid unwanted column names in the resulting Data Frame, always provide aliases for all returned properties.
|
|
762
768
|
|
|
763
769
|
```python
|
|
764
770
|
query = """
|
|
@@ -768,35 +774,56 @@ query = """
|
|
|
768
774
|
m.tagline AS tagline;
|
|
769
775
|
"""
|
|
770
776
|
|
|
771
|
-
data = graph.execute_read_query(query)
|
|
777
|
+
data = graph.execute_read_query(query,db_info['database'])
|
|
772
778
|
```
|
|
779
|
+
|
|
780
|
+
### Submitting a Cypher read query with parameters
|
|
781
|
+
|
|
782
|
+
The method execute\_read\_query can optionally received a Python Dictionary with the parameters required as part of you Cypher query.
|
|
783
|
+
|
|
784
|
+
```python
|
|
785
|
+
query = """
|
|
786
|
+
MATCH (m:Movie {title: $movie_title})
|
|
787
|
+
RETURN m.tile AS title,
|
|
788
|
+
m.released AS releasedYear,
|
|
789
|
+
m.tagline AS tagline;
|
|
790
|
+
"""
|
|
791
|
+
|
|
792
|
+
data = graph.execute_read_query(query, db_info['database'], db_info['database'], parameters={'movie_title':'The Matrix'})
|
|
793
|
+
```
|
|
794
|
+
|
|
773
795
|
### Submitting a Cypher write query
|
|
774
796
|
|
|
775
|
-
|
|
797
|
+
You can submit a Cypher write query using the method execute_write_query. The signature of this method is identical to the execute_read_query. This method returns a dictionary containing the updates made to the database. As with read queries you can also specify a Python dictionary containing Cypher parameters.
|
|
776
798
|
|
|
777
799
|
```python
|
|
778
800
|
query = """
|
|
779
|
-
MERGE(m:Movie {title
|
|
801
|
+
MERGE(m:Movie {title:$movie_title})
|
|
780
802
|
ON CREATE SET
|
|
781
|
-
m.release=
|
|
782
|
-
m.tagline =
|
|
803
|
+
m.release = $year,
|
|
804
|
+
m.tagline = $tagline
|
|
783
805
|
"""
|
|
784
806
|
|
|
785
|
-
results = graph.execute_write_query(query
|
|
807
|
+
results = graph.execute_write_query(query, db_info['database'],
|
|
808
|
+
parameters={
|
|
809
|
+
'movie_title':'Wakanda Forever',
|
|
810
|
+
'year'=2022,
|
|
811
|
+
tagline=''Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!''
|
|
812
|
+
})
|
|
786
813
|
```
|
|
787
814
|
|
|
788
815
|
### Updating a database with data in a DataFrame: Sequentially
|
|
789
816
|
|
|
790
|
-
We can update a Neo4j database with data contained in a Pandas
|
|
817
|
+
We can update a Neo4j database with data contained in a Pandas Data Frame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methods return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
791
818
|
|
|
792
|
-
- _data_: Pandas
|
|
793
|
-
- _database_: Optional, database on which to execute the
|
|
794
|
-
- _partitions_: Optional, number of partitions use to break the
|
|
795
|
-
- _parallel_:
|
|
819
|
+
- _data_: Pandas Data Frame containing data to process.
|
|
820
|
+
- _database_: Optional, database on which to execute the queries.
|
|
821
|
+
- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
|
|
822
|
+
- _parallel_: Whether to execute the query using multiple Python process.
|
|
796
823
|
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
797
824
|
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
798
825
|
|
|
799
|
-
To illustrates how to use
|
|
826
|
+
To illustrates how to use these methods lets imagine we have a csv file that looks like this:
|
|
800
827
|
|
|
801
828
|
```python
|
|
802
829
|
# people.csv
|
|
@@ -812,11 +839,11 @@ personId,name,birthYear,deathYear
|
|
|
812
839
|
3482,Chus Lampreave,1930,
|
|
813
840
|
```
|
|
814
841
|
|
|
815
|
-
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to
|
|
842
|
+
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performed the following steps:
|
|
816
843
|
|
|
817
|
-
1. Load the csv to Python as a
|
|
818
|
-
2. Develop the Cypher query that is going to be use to load the
|
|
819
|
-
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the
|
|
844
|
+
1. Load the csv to Python as a Data Frame.
|
|
845
|
+
2. Develop the Cypher query that is going to be use to load the Data Frame into Neo4j.
|
|
846
|
+
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the Data Frame, Cypher query and any other optional arguments to this method.
|
|
820
847
|
|
|
821
848
|
```python
|
|
822
849
|
import pandas as pd
|
|
@@ -834,10 +861,10 @@ person_load_query = """
|
|
|
834
861
|
p.deathYear = toInteger(row.deathYear)
|
|
835
862
|
"""
|
|
836
863
|
|
|
837
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
864
|
+
person_load_result = graph.execute_write_query_with_data(person_load_query, person_data, db_info['database'])
|
|
838
865
|
```
|
|
839
866
|
|
|
840
|
-
|
|
867
|
+
It is not a good practice to store your Cypher queries as strings within your Python modules. To solve this, we can extend our YAML configuration file to store all our Cypher queries.
|
|
841
868
|
|
|
842
869
|
```yaml
|
|
843
870
|
# configuration.yaml
|
|
@@ -845,9 +872,10 @@ db_info:
|
|
|
845
872
|
uri: neo4j_uri
|
|
846
873
|
user: neo4j_user
|
|
847
874
|
password: user_password
|
|
875
|
+
database: neo4j
|
|
848
876
|
loading_queries:
|
|
849
877
|
nodes:
|
|
850
|
-
|
|
878
|
+
Person: |
|
|
851
879
|
WITH $rows AS rows
|
|
852
880
|
UNWIND rows AS row
|
|
853
881
|
MERGE(p:Person {id:row.personId})
|
|
@@ -867,28 +895,33 @@ person_data = pd.read_csv('people.csv')
|
|
|
867
895
|
|
|
868
896
|
node_load_queries = config['loading_queries']['nodes']
|
|
869
897
|
|
|
870
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
898
|
+
person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
|
|
871
899
|
```
|
|
872
900
|
|
|
873
|
-
If the number of rows in your
|
|
901
|
+
If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
874
902
|
|
|
875
903
|
```python
|
|
876
904
|
person_load_result = graph.execute_write_query_with_data(
|
|
877
|
-
person_data,
|
|
878
905
|
node_load_queries['person'],
|
|
906
|
+
person_data,
|
|
907
|
+
db_info['database'],
|
|
879
908
|
partitions = 5
|
|
880
909
|
)
|
|
881
910
|
```
|
|
882
|
-
## Updating a database with data in a
|
|
911
|
+
## Updating a database with data in a Data Frame: Parallel
|
|
883
912
|
|
|
884
|
-
To load the data in the
|
|
913
|
+
To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
|
|
885
914
|
|
|
886
915
|
```python
|
|
887
916
|
person_load_result = graph.execute_write_query_with_data(
|
|
888
|
-
person_data,
|
|
889
917
|
node_load_queries['person'],
|
|
918
|
+
person_data,
|
|
890
919
|
partitions = 5,
|
|
891
920
|
parallel = True,
|
|
892
921
|
workers = 5
|
|
893
922
|
)
|
|
894
923
|
```
|
|
924
|
+
|
|
925
|
+
## More detailed examples
|
|
926
|
+
|
|
927
|
+
For more detailed examples please referred to this [article](https://medium.com/@a-gilmore/pyneoinstance-a-user-friendly-python-library-for-neo4j-dbefa3117bb2).
|
|
@@ -6,10 +6,10 @@ build-backend = "setuptools.build_meta"
|
|
|
6
6
|
|
|
7
7
|
[project]
|
|
8
8
|
name = "pyneoinstance"
|
|
9
|
-
version = "1.1.
|
|
9
|
+
version = "1.1.2"
|
|
10
10
|
description = "Handle Neo4j read and write transactions"
|
|
11
11
|
readme = "README.md"
|
|
12
|
-
authors = [{ name = "Jose A Alvarado-
|
|
12
|
+
authors = [{ name = "Jose A. Alvarado-Guzman",email = "jose.alvarado-guzman@neo4j.com" }]
|
|
13
13
|
license = { file = "LICENSE" }
|
|
14
14
|
classifiers = [
|
|
15
15
|
"License :: OSI Approved :: GNU General Public License (GPL)",
|
|
@@ -4,8 +4,7 @@ pre_ingest:
|
|
|
4
4
|
- CREATE CONSTRAINT nkMovie IF NOT EXISTS FOR(n:Movie) REQUIRE (n.movieId) IS NODE KEY;
|
|
5
5
|
|
|
6
6
|
queries:
|
|
7
|
-
load_person:
|
|
8
|
-
|
|
|
7
|
+
load_person: |
|
|
9
8
|
WITH $rows AS rows
|
|
10
9
|
UNWIND rows AS row
|
|
11
10
|
MERGE(p:Person {personId:row.personId})
|
|
@@ -13,8 +12,7 @@ queries:
|
|
|
13
12
|
SET p.personId=row.personId,
|
|
14
13
|
p.name=row.name,
|
|
15
14
|
p.birthYear=toInteger(row.birthYear)
|
|
16
|
-
load_movie:
|
|
17
|
-
|
|
|
15
|
+
load_movie: |
|
|
18
16
|
WITH $rows AS rows
|
|
19
17
|
UNWIND rows AS row
|
|
20
18
|
MERGE(m:Movie {movieId:row.movieId})
|
|
@@ -24,11 +22,16 @@ queries:
|
|
|
24
22
|
m.averageVote=toFloat(row.avgVote),
|
|
25
23
|
m.releaseYear=toInteger(row.releaseYear),
|
|
26
24
|
m.genres=split(row.genres,':')
|
|
27
|
-
load_role:
|
|
28
|
-
|
|
|
25
|
+
load_role: |
|
|
29
26
|
WITH $rows AS rows
|
|
30
27
|
UNWIND rows AS row
|
|
31
28
|
MATCH(p:Person {personId:row.personId}),(m:Movie {movieId:row.movieId})
|
|
32
29
|
MERGE(p)-[r:ACTED_IN]->(m)
|
|
33
30
|
ON CREATE
|
|
34
31
|
SET r.character=split(row.character,':')
|
|
32
|
+
create_actor: |
|
|
33
|
+
MERGE(p:Person:Actor {personId:$id})
|
|
34
|
+
ON CREATE
|
|
35
|
+
SET p.personId=$id,
|
|
36
|
+
p.name=$name,
|
|
37
|
+
p.birthYear=$year
|
|
@@ -93,11 +93,21 @@ class TestNeo4jInstance(unittest.TestCase):
|
|
|
93
93
|
# Test the execute_read_query_using cypher parameters
|
|
94
94
|
movie = "The Matrix"
|
|
95
95
|
query = "MATCH(m:Movie) WHERE m.title=$movie_name RETURN m.title AS movie"
|
|
96
|
-
print('Parameters')
|
|
96
|
+
print('Read Parameters')
|
|
97
97
|
result = self.graph.execute_read_query(query,
|
|
98
98
|
self.queries['database'],
|
|
99
|
-
movie_name
|
|
99
|
+
{'movie_name':movie})
|
|
100
100
|
self.assertEqual(movie,result['movie'][0])
|
|
101
101
|
|
|
102
|
+
# Test the execute_write_query_using cypher parameters
|
|
103
|
+
print('Write Parameters')
|
|
104
|
+
result = self.graph.execute_write_query(
|
|
105
|
+
self.queries['queries']['create_actor'],
|
|
106
|
+
self.queries['database'],
|
|
107
|
+
parameters={'id':-1,'name':'Darth Vader','year':'1977'})
|
|
108
|
+
solution = {'nodes_created':1,'labels_added':2,
|
|
109
|
+
'properties_set':4}
|
|
110
|
+
self.assertEqual(solution, result)
|
|
111
|
+
|
|
102
112
|
if __name__ == '__main__':
|
|
103
113
|
unittest.main()
|
pyneoinstance-1.1.0/README.md
DELETED
|
@@ -1,206 +0,0 @@
|
|
|
1
|
-
# PyNeoInstance Package
|
|
2
|
-
|
|
3
|
-
## What is it?
|
|
4
|
-
|
|
5
|
-
Is a Python package that provides a user friendly API for submitting Cypher queries to Neo4j.
|
|
6
|
-
It supports the following features:
|
|
7
|
-
|
|
8
|
-
- __Read query__: Submit a Cypher query and get the result of it as a Pandas DataFrame.
|
|
9
|
-
- __Write query__: Sumbmit a Cypher query to update an existing database.
|
|
10
|
-
- __Write query with data__: Update a database based on a Pandas DataFrame and one or more Cypher queries.
|
|
11
|
-
|
|
12
|
-
## How to install it?
|
|
13
|
-
|
|
14
|
-
```python
|
|
15
|
-
pip install pyneoinstance
|
|
16
|
-
```
|
|
17
|
-
|
|
18
|
-
## How to use it?
|
|
19
|
-
|
|
20
|
-
### Instantiating the Neo4jInstance class
|
|
21
|
-
|
|
22
|
-
In order to use this package you first need to import the Neo4jInstance class.
|
|
23
|
-
|
|
24
|
-
```python
|
|
25
|
-
from pyneoingest import Neo4jInstance
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
Once imported, you need to instantiate this class by providing the following arguments:
|
|
29
|
-
|
|
30
|
-
- _uri_: The Neo4j instance unique resource identifier.
|
|
31
|
-
- _user_: The user name use to authenticate.
|
|
32
|
-
- _password_: The user name password.
|
|
33
|
-
|
|
34
|
-
```python
|
|
35
|
-
uri = 'bolt://localhost:7687'
|
|
36
|
-
user = 'neo4j'
|
|
37
|
-
password = 'password'
|
|
38
|
-
|
|
39
|
-
graph = Neo4jInstance(uri, user, password)
|
|
40
|
-
```
|
|
41
|
-
|
|
42
|
-
Although the previous code is valid, it is not recommended to hard code your user credentials in your application. As an alternative you can store this information as part of your OS user environment variables and then retrieve them in Python.
|
|
43
|
-
|
|
44
|
-
```python
|
|
45
|
-
import os
|
|
46
|
-
|
|
47
|
-
uri = os.environ['NEO4J_URI']
|
|
48
|
-
user = os.environ['NEO4J_USER']
|
|
49
|
-
password = os.environ['NEO4J_PASSWORD']
|
|
50
|
-
graph = Neo4jInstance(uri,user,password)
|
|
51
|
-
```
|
|
52
|
-
Another alternative is to create a YAML configutation file to store this infromation.
|
|
53
|
-
|
|
54
|
-
```yaml
|
|
55
|
-
# configuration.yaml
|
|
56
|
-
db_info:
|
|
57
|
-
uri: neo4j_uri
|
|
58
|
-
user: neo4j_user
|
|
59
|
-
password: user_password
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
This YAML file can then be loaded in Python and these information can be retrieve.
|
|
63
|
-
|
|
64
|
-
```python
|
|
65
|
-
from pyneoinstance import load_yaml_file
|
|
66
|
-
|
|
67
|
-
config = load_yaml_file(configuration.yaml)
|
|
68
|
-
db_info = config['db_info']
|
|
69
|
-
graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
70
|
-
```
|
|
71
|
-
### Submitting a Cypher read query
|
|
72
|
-
|
|
73
|
-
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receive the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query is return as a a Pandas DataFrame. To avoid unwanted column names in the resulting DataFrame, always provide aliases for all return properties.
|
|
74
|
-
|
|
75
|
-
```python
|
|
76
|
-
query = """
|
|
77
|
-
MATCH (m:Movie {title: "The Matrix"})
|
|
78
|
-
RETURN m.tile AS title,
|
|
79
|
-
m.released AS releasedYear,
|
|
80
|
-
m.tagline AS tagline;
|
|
81
|
-
"""
|
|
82
|
-
|
|
83
|
-
data = graph.execute_read_query(query)
|
|
84
|
-
```
|
|
85
|
-
### Submitting a Cypher write query
|
|
86
|
-
|
|
87
|
-
We submit a Cypher write query using the method execute\_write\_query. The signature of this method is identical to the execute\_read\_query. This method returns a dictionary containing the updates made to the database.
|
|
88
|
-
|
|
89
|
-
```python
|
|
90
|
-
query = """
|
|
91
|
-
MERGE(m:Movie {title:'Wakanda Forever'})
|
|
92
|
-
ON CREATE SET
|
|
93
|
-
m.release=2022,
|
|
94
|
-
m.tagline = 'Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!'
|
|
95
|
-
"""
|
|
96
|
-
|
|
97
|
-
results = graph.execute_write_query(query)
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
### Updating a database with data in a DataFrame: Sequentially
|
|
101
|
-
|
|
102
|
-
We can update a Neo4j database with data contained in a Pandas DataFrame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methos return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
103
|
-
|
|
104
|
-
- _data_: Pandas DataFrame containing data to process.
|
|
105
|
-
- _database_: Optional, database on which to execute the quieries.
|
|
106
|
-
- _partitions_: Optional, number of partitions use to break the DataFrame in smaller chucks.
|
|
107
|
-
- _parallel_: Wheather to execute the query using multiple Python process.
|
|
108
|
-
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
109
|
-
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
110
|
-
|
|
111
|
-
To illustrates how to use this methods lets imagine we have a csv file that looks like this:
|
|
112
|
-
|
|
113
|
-
```python
|
|
114
|
-
# people.csv
|
|
115
|
-
personId,name,birthYear,deathYear
|
|
116
|
-
23945,Gérard Pirès,1942,
|
|
117
|
-
553509,Helen Reddy,1941,
|
|
118
|
-
113934,Susan Flannery,1939,
|
|
119
|
-
26706,David Rintoul,1948,
|
|
120
|
-
237537,Rita Marley,1946,
|
|
121
|
-
11502,Harry Davenport,1866,1949
|
|
122
|
-
11792,George Dickerson,1933,
|
|
123
|
-
7242,Temuera Morrison,1960,
|
|
124
|
-
3482,Chus Lampreave,1930,
|
|
125
|
-
```
|
|
126
|
-
|
|
127
|
-
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performe the following steps:
|
|
128
|
-
|
|
129
|
-
1. Load the csv to Python as a DataFrame.
|
|
130
|
-
2. Develop the Cypher query that is going to be use to load the DataFrame into Neo4j.
|
|
131
|
-
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the DataFrame, Cypher query and any other optinal arguments to this method.
|
|
132
|
-
|
|
133
|
-
```python
|
|
134
|
-
import pandas as pd
|
|
135
|
-
|
|
136
|
-
person_data = pd.read_csv('people.csv')
|
|
137
|
-
|
|
138
|
-
person_load_query = """
|
|
139
|
-
WITH $rows AS rows
|
|
140
|
-
UNWIND rows AS row
|
|
141
|
-
MERGE(p:Person {id:row.personId})
|
|
142
|
-
ON CREATE
|
|
143
|
-
SET p.id = row.personId,
|
|
144
|
-
p.name = row.name,
|
|
145
|
-
p.birthYear=toInteger(row.birthYear),
|
|
146
|
-
p.deathYear = toInteger(row.deathYear)
|
|
147
|
-
"""
|
|
148
|
-
|
|
149
|
-
person_load_result = graph.execute_write_query_with_data(person_data, person_load_query)
|
|
150
|
-
```
|
|
151
|
-
|
|
152
|
-
Is not a good practice to store your Cypher queries as strings within your Python modules. To solve this we can extend our YAML configuration file to store all our Cypher queries.
|
|
153
|
-
|
|
154
|
-
```yaml
|
|
155
|
-
# configuration.yaml
|
|
156
|
-
db_info:
|
|
157
|
-
uri: neo4j_uri
|
|
158
|
-
user: neo4j_user
|
|
159
|
-
password: user_password
|
|
160
|
-
loading_queries:
|
|
161
|
-
nodes:
|
|
162
|
-
person: |
|
|
163
|
-
WITH $rows AS rows
|
|
164
|
-
UNWIND rows AS row
|
|
165
|
-
MERGE(p:Person {id:row.personId})
|
|
166
|
-
ON CREATE
|
|
167
|
-
SET p.id = row.personId,
|
|
168
|
-
p.name = row.name,
|
|
169
|
-
p.birthYear=toInteger(row.birthYear),
|
|
170
|
-
p.deathYear = toInteger(row.deathYear)
|
|
171
|
-
```
|
|
172
|
-
|
|
173
|
-
Now that we added the loading query to our configuration file, lets re-write our Python script to get this query from the configuration file.
|
|
174
|
-
|
|
175
|
-
```python
|
|
176
|
-
import pandas as pd
|
|
177
|
-
|
|
178
|
-
person_data = pd.read_csv('people.csv')
|
|
179
|
-
|
|
180
|
-
node_load_queries = config['loading_queries']['nodes']
|
|
181
|
-
|
|
182
|
-
person_load_result = graph.execute_write_query_with_data(person_data, node_load_queries['person'])
|
|
183
|
-
```
|
|
184
|
-
|
|
185
|
-
If the number of rows in your DataFrame is 100K or more it is reccomended to break the DataFrame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this DataFrame into. This partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
186
|
-
|
|
187
|
-
```python
|
|
188
|
-
person_load_result = graph.execute_write_query_with_data(
|
|
189
|
-
person_data,
|
|
190
|
-
node_load_queries['person'],
|
|
191
|
-
partitions = 5
|
|
192
|
-
)
|
|
193
|
-
```
|
|
194
|
-
## Updating a database with data in a DataFrame: Parallel
|
|
195
|
-
|
|
196
|
-
To load the data in the DataFrame in parallel, we need to partition the DataFrame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default this will use all the available virtual CPUs in the machine. If you need to use less, you can specified how many to use by setting the _workers_ argument.
|
|
197
|
-
|
|
198
|
-
```python
|
|
199
|
-
person_load_result = graph.execute_write_query_with_data(
|
|
200
|
-
person_data,
|
|
201
|
-
node_load_queries['person'],
|
|
202
|
-
partitions = 5,
|
|
203
|
-
parallel = True,
|
|
204
|
-
workers = 5
|
|
205
|
-
)
|
|
206
|
-
```
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|