pyneoinstance 1.1.0__tar.gz → 2.0.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/MANIFEST.in +1 -0
- {pyneoinstance-1.1.0/pyneoinstance.egg-info → pyneoinstance-2.0.2}/PKG-INFO +144 -38
- pyneoinstance-2.0.2/README.md +308 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/__init__.py +1 -1
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/database/neo4jdbms.py +403 -15
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2/pyneoinstance.egg-info}/PKG-INFO +144 -38
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance.egg-info/SOURCES.txt +2 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance.egg-info/requires.txt +1 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance.egg-info/top_level.txt +1 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyproject.toml +5 -4
- pyneoinstance-2.0.2/setup.py +5 -0
- pyneoinstance-2.0.2/test/__init__.py +3 -0
- pyneoinstance-2.0.2/test/context.py +8 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/test/cypher_queries.yaml +10 -7
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/test/test_neo4jdb.py +101 -3
- pyneoinstance-1.1.0/README.md +0 -206
- pyneoinstance-1.1.0/setup.py +0 -3
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/LICENSE +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/database/__init__.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/database/context.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/fileload/__init__.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/fileload/context.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/fileload/helpers.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/fileload/loadyaml.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/util/__init__.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance/util/functions.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/pyneoinstance.egg-info/dependency_links.txt +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/setup.cfg +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/test/load_files.yaml +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/test/movies.csv +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/test/people.csv +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/test/roles.csv +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/test/test.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/test/test_fileload.py +0 -0
- {pyneoinstance-1.1.0 → pyneoinstance-2.0.2}/test/test_util_functions.py +0 -0
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: pyneoinstance
|
|
3
|
-
Version:
|
|
4
|
-
Summary: Handle Neo4j read and write transactions
|
|
5
|
-
Author-email: Jose A Alvarado-
|
|
3
|
+
Version: 2.0.2
|
|
4
|
+
Summary: Handle Neo4j read and write transactions and provide methos for performing graph EDA
|
|
5
|
+
Author-email: "Jose A. Alvarado-Guzman" <jose.alvarado-guzman@neo4j.com>
|
|
6
6
|
License: GNU GENERAL PUBLIC LICENSE
|
|
7
7
|
Version 3, 29 June 2007
|
|
8
8
|
|
|
@@ -685,17 +685,33 @@ Classifier: Development Status :: 4 - Beta
|
|
|
685
685
|
Requires-Python: >=3.8
|
|
686
686
|
Description-Content-Type: text/markdown
|
|
687
687
|
License-File: LICENSE
|
|
688
|
+
Requires-Dist: neo4j>=5.0.0
|
|
689
|
+
Requires-Dist: pandas
|
|
690
|
+
Requires-Dist: PyYAML
|
|
691
|
+
Requires-Dist: pyvis
|
|
688
692
|
|
|
689
693
|
# PyNeoInstance Package
|
|
690
694
|
|
|
691
695
|
## What is it?
|
|
692
696
|
|
|
693
|
-
|
|
697
|
+
PyNeoInstance is a Python package that provides a user friendly API for submitting Cypher queries to Neo4j and performing Exploratory Data Analysis (EDA) of your graph.
|
|
698
|
+
|
|
694
699
|
It supports the following features:
|
|
695
700
|
|
|
696
|
-
-
|
|
697
|
-
-
|
|
698
|
-
-
|
|
701
|
+
- __<u>Read query</u>__: Submit a Cypher read query, with or without parameters, and get a Pandas Data Frame with the results.
|
|
702
|
+
- __<u>Write query</u>__: Submit a write Cypher query, with or without parameters, to update an existing database.
|
|
703
|
+
- __<u>Write queries</u>__: Submit a list of Cypher queries to update an existing database.
|
|
704
|
+
- __<u>Write query with data</u>__: Update a database based on a Pandas Data Frame and a Cypher query.
|
|
705
|
+
- __<u>Write queries with data</u>__: Update a database based on a Pandas Data Frame and a list of Cypher queries.
|
|
706
|
+
- __<u>Node label frequency</u>__: Get the distribution of the graph node labels.
|
|
707
|
+
- __<u>Multi label node frequency</u>__: Get the distribution of the graph multi label nodes.
|
|
708
|
+
- __<u>Relationship type frequency</u>__: Get the distribution of the graph relationship types.
|
|
709
|
+
- __<u>Relationship type source-target frequency</u>__: Get the distribution of relationship source and target.
|
|
710
|
+
- __<u>Node and relationship properties</u>__: Get information of all properties in the graph.
|
|
711
|
+
- __<u>Constraints</u>__: Get information of all constraints in the graph.
|
|
712
|
+
- __<u>Indexes</u>__: Get information of all indexes in the graph.
|
|
713
|
+
- __<u>Schema Visualization</u>__: Visualize the graph schema.
|
|
714
|
+
|
|
699
715
|
|
|
700
716
|
## How to install it?
|
|
701
717
|
|
|
@@ -707,16 +723,16 @@ pip install pyneoinstance
|
|
|
707
723
|
|
|
708
724
|
### Instantiating the Neo4jInstance class
|
|
709
725
|
|
|
710
|
-
In order to use this package you first need to import the Neo4jInstance class.
|
|
726
|
+
In order to use this package, you first need to import the Neo4jInstance class.
|
|
711
727
|
|
|
712
728
|
```python
|
|
713
|
-
from
|
|
729
|
+
from pyneoinstance import Neo4jInstance
|
|
714
730
|
```
|
|
715
731
|
|
|
716
732
|
Once imported, you need to instantiate this class by providing the following arguments:
|
|
717
733
|
|
|
718
734
|
- _uri_: The Neo4j instance unique resource identifier.
|
|
719
|
-
- _user_: The user name
|
|
735
|
+
- _user_: The user name used to authenticate.
|
|
720
736
|
- _password_: The user name password.
|
|
721
737
|
|
|
722
738
|
```python
|
|
@@ -737,7 +753,7 @@ user = os.environ['NEO4J_USER']
|
|
|
737
753
|
password = os.environ['NEO4J_PASSWORD']
|
|
738
754
|
graph = Neo4jInstance(uri,user,password)
|
|
739
755
|
```
|
|
740
|
-
Another alternative is to create a YAML
|
|
756
|
+
Another alternative is to create a YAML configuration file to store this information.
|
|
741
757
|
|
|
742
758
|
```yaml
|
|
743
759
|
# configuration.yaml
|
|
@@ -745,9 +761,10 @@ db_info:
|
|
|
745
761
|
uri: neo4j_uri
|
|
746
762
|
user: neo4j_user
|
|
747
763
|
password: user_password
|
|
764
|
+
database: neo4j
|
|
748
765
|
```
|
|
749
766
|
|
|
750
|
-
This YAML file can then be loaded in Python and
|
|
767
|
+
This YAML file can then be loaded in Python and this information can be retrieve.
|
|
751
768
|
|
|
752
769
|
```python
|
|
753
770
|
from pyneoinstance import load_yaml_file
|
|
@@ -758,7 +775,7 @@ graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
|
758
775
|
```
|
|
759
776
|
### Submitting a Cypher read query
|
|
760
777
|
|
|
761
|
-
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method
|
|
778
|
+
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receives the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query are return as a Pandas Data Frame. To avoid unwanted column names in the resulting Data Frame, always provide aliases for all returned properties.
|
|
762
779
|
|
|
763
780
|
```python
|
|
764
781
|
query = """
|
|
@@ -768,35 +785,56 @@ query = """
|
|
|
768
785
|
m.tagline AS tagline;
|
|
769
786
|
"""
|
|
770
787
|
|
|
771
|
-
data = graph.execute_read_query(query)
|
|
788
|
+
data = graph.execute_read_query(query,db_info['database'])
|
|
772
789
|
```
|
|
790
|
+
|
|
791
|
+
### Submitting a Cypher read query with parameters
|
|
792
|
+
|
|
793
|
+
The method execute\_read\_query can optionally received a Python Dictionary with the parameters required as part of you Cypher query.
|
|
794
|
+
|
|
795
|
+
```python
|
|
796
|
+
query = """
|
|
797
|
+
MATCH (m:Movie {title: $movie_title})
|
|
798
|
+
RETURN m.tile AS title,
|
|
799
|
+
m.released AS releasedYear,
|
|
800
|
+
m.tagline AS tagline;
|
|
801
|
+
"""
|
|
802
|
+
|
|
803
|
+
data = graph.execute_read_query(query, db_info['database'], db_info['database'], parameters={'movie_title':'The Matrix'})
|
|
804
|
+
```
|
|
805
|
+
|
|
773
806
|
### Submitting a Cypher write query
|
|
774
807
|
|
|
775
|
-
|
|
808
|
+
You can submit a Cypher write query using the method execute_write_query. The signature of this method is identical to the execute_read_query. This method returns a dictionary containing the updates made to the database. As with read queries you can also specify a Python dictionary containing Cypher parameters.
|
|
776
809
|
|
|
777
810
|
```python
|
|
778
811
|
query = """
|
|
779
|
-
MERGE(m:Movie {title
|
|
812
|
+
MERGE(m:Movie {title:$movie_title})
|
|
780
813
|
ON CREATE SET
|
|
781
|
-
m.release=
|
|
782
|
-
m.tagline =
|
|
814
|
+
m.release = $year,
|
|
815
|
+
m.tagline = $tagline
|
|
783
816
|
"""
|
|
784
817
|
|
|
785
|
-
results = graph.execute_write_query(query
|
|
818
|
+
results = graph.execute_write_query(query, db_info['database'],
|
|
819
|
+
parameters={
|
|
820
|
+
'movie_title':'Wakanda Forever',
|
|
821
|
+
'year'=2022,
|
|
822
|
+
tagline=''Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!''
|
|
823
|
+
})
|
|
786
824
|
```
|
|
787
825
|
|
|
788
826
|
### Updating a database with data in a DataFrame: Sequentially
|
|
789
827
|
|
|
790
|
-
We can update a Neo4j database with data contained in a Pandas
|
|
828
|
+
We can update a Neo4j database with data contained in a Pandas Data Frame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methods return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
791
829
|
|
|
792
|
-
- _data_: Pandas
|
|
793
|
-
- _database_: Optional, database on which to execute the
|
|
794
|
-
- _partitions_: Optional, number of partitions use to break the
|
|
795
|
-
- _parallel_:
|
|
830
|
+
- _data_: Pandas Data Frame containing data to process.
|
|
831
|
+
- _database_: Optional, database on which to execute the queries.
|
|
832
|
+
- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
|
|
833
|
+
- _parallel_: Whether to execute the query using multiple Python process.
|
|
796
834
|
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
797
835
|
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
798
836
|
|
|
799
|
-
To illustrates how to use
|
|
837
|
+
To illustrates how to use these methods lets imagine we have a csv file that looks like this:
|
|
800
838
|
|
|
801
839
|
```python
|
|
802
840
|
# people.csv
|
|
@@ -812,11 +850,11 @@ personId,name,birthYear,deathYear
|
|
|
812
850
|
3482,Chus Lampreave,1930,
|
|
813
851
|
```
|
|
814
852
|
|
|
815
|
-
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to
|
|
853
|
+
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performed the following steps:
|
|
816
854
|
|
|
817
|
-
1. Load the csv to Python as a
|
|
818
|
-
2. Develop the Cypher query that is going to be use to load the
|
|
819
|
-
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the
|
|
855
|
+
1. Load the csv to Python as a Data Frame.
|
|
856
|
+
2. Develop the Cypher query that is going to be use to load the Data Frame into Neo4j.
|
|
857
|
+
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the Data Frame, Cypher query and any other optional arguments to this method.
|
|
820
858
|
|
|
821
859
|
```python
|
|
822
860
|
import pandas as pd
|
|
@@ -834,10 +872,10 @@ person_load_query = """
|
|
|
834
872
|
p.deathYear = toInteger(row.deathYear)
|
|
835
873
|
"""
|
|
836
874
|
|
|
837
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
875
|
+
person_load_result = graph.execute_write_query_with_data(person_load_query, person_data, db_info['database'])
|
|
838
876
|
```
|
|
839
877
|
|
|
840
|
-
|
|
878
|
+
It is not a good practice to store your Cypher queries as strings within your Python modules. To solve this, we can extend our YAML configuration file to store all our Cypher queries.
|
|
841
879
|
|
|
842
880
|
```yaml
|
|
843
881
|
# configuration.yaml
|
|
@@ -845,9 +883,10 @@ db_info:
|
|
|
845
883
|
uri: neo4j_uri
|
|
846
884
|
user: neo4j_user
|
|
847
885
|
password: user_password
|
|
886
|
+
database: neo4j
|
|
848
887
|
loading_queries:
|
|
849
888
|
nodes:
|
|
850
|
-
|
|
889
|
+
Person: |
|
|
851
890
|
WITH $rows AS rows
|
|
852
891
|
UNWIND rows AS row
|
|
853
892
|
MERGE(p:Person {id:row.personId})
|
|
@@ -867,28 +906,95 @@ person_data = pd.read_csv('people.csv')
|
|
|
867
906
|
|
|
868
907
|
node_load_queries = config['loading_queries']['nodes']
|
|
869
908
|
|
|
870
|
-
person_load_result = graph.execute_write_query_with_data(person_data,
|
|
909
|
+
person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
|
|
871
910
|
```
|
|
872
911
|
|
|
873
|
-
If the number of rows in your
|
|
912
|
+
If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
874
913
|
|
|
875
914
|
```python
|
|
876
915
|
person_load_result = graph.execute_write_query_with_data(
|
|
877
|
-
person_data,
|
|
878
916
|
node_load_queries['person'],
|
|
917
|
+
person_data,
|
|
918
|
+
db_info['database'],
|
|
879
919
|
partitions = 5
|
|
880
920
|
)
|
|
881
921
|
```
|
|
882
|
-
## Updating a database with data in a
|
|
922
|
+
## Updating a database with data in a Data Frame: Parallel
|
|
883
923
|
|
|
884
|
-
To load the data in the
|
|
924
|
+
To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
|
|
885
925
|
|
|
886
926
|
```python
|
|
887
927
|
person_load_result = graph.execute_write_query_with_data(
|
|
888
|
-
person_data,
|
|
889
928
|
node_load_queries['person'],
|
|
929
|
+
person_data,
|
|
890
930
|
partitions = 5,
|
|
891
931
|
parallel = True,
|
|
892
932
|
workers = 5
|
|
893
933
|
)
|
|
894
934
|
```
|
|
935
|
+
## Graph EDA
|
|
936
|
+
|
|
937
|
+
### Node Label Frequency
|
|
938
|
+
The get_node_label_freq method provides the frequency and relative frequency of all nodes in the graph. This method consider all node labels as single labels, if you want the distribution considering multiple labels use the get_node_multilabel_freq method.
|
|
939
|
+
```python
|
|
940
|
+
graph.get_node_label_freq(database='mydbname')
|
|
941
|
+
```
|
|
942
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/nodefreq.png" width="250">
|
|
943
|
+
|
|
944
|
+
### Relationship Type Frequency
|
|
945
|
+
The get_rela_type_freq method provides the frequency and relative frequency of all relationship types in the graph.
|
|
946
|
+
```python
|
|
947
|
+
graph.get_rela_type_freq(database='mydbname')
|
|
948
|
+
```
|
|
949
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/relaFreq.png" width="300">
|
|
950
|
+
|
|
951
|
+
### Relationship Source-Target Frequency
|
|
952
|
+
The get_rela_source_target_freq method provides the frequency and relative frequency of all relationship types considering the node label of the source and targets of the relationships.
|
|
953
|
+
```python
|
|
954
|
+
graph.get_rela_type_freq(database='mydbname')
|
|
955
|
+
```
|
|
956
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/relaSourceTargetFreq.png" width="300">
|
|
957
|
+
|
|
958
|
+
### Properties
|
|
959
|
+
The get_properties method provides information about all the nodes and relationships properties.
|
|
960
|
+
```python
|
|
961
|
+
graph.graph.get_properties(database='mydbname')
|
|
962
|
+
```
|
|
963
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/properties.png" width="350" hight="300">
|
|
964
|
+
|
|
965
|
+
### Constraints
|
|
966
|
+
The get_constraints method provides information about all the graph constraints.
|
|
967
|
+
```python
|
|
968
|
+
graph.get_constraints(database='mydbname')
|
|
969
|
+
```
|
|
970
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/constraints.png" width="350" hight="300">
|
|
971
|
+
|
|
972
|
+
### Indexes
|
|
973
|
+
The get_indexes method provides information about all the graph indexes.
|
|
974
|
+
```python
|
|
975
|
+
graph.get_indexes(database='mydbname')
|
|
976
|
+
```
|
|
977
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/indexes.png" width="450" hight="400">
|
|
978
|
+
|
|
979
|
+
### Schema Visualization
|
|
980
|
+
The graph.get_schema_visualization provides an interactive UI with the visualization of the graph schema with the ability to filter the schema. This UI can be access via a Notebook or HTML page. This visualization displays the node labels as string close to nodes, relationship types are only display on mouse hover.
|
|
981
|
+
|
|
982
|
+
```python
|
|
983
|
+
schema = graph.get_schema_visualization(database='mydbname')
|
|
984
|
+
schema.show('schema.html')
|
|
985
|
+
```
|
|
986
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/schema.png" width="450" hight="400">
|
|
987
|
+
|
|
988
|
+
To access the UI via HTTML page just set the parameter notebook to false in the show method and open the html file in your browser.
|
|
989
|
+
|
|
990
|
+
```python
|
|
991
|
+
schema = graph.get_schema_visualization(database='mydbname')
|
|
992
|
+
schema.show('schema.html',notebook=False)
|
|
993
|
+
```
|
|
994
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/schemaHtml.png" width="450" hight="400">
|
|
995
|
+
|
|
996
|
+
## More detailed examples
|
|
997
|
+
|
|
998
|
+
For more detailed examples please refer to the following articles:
|
|
999
|
+
* Data ingestion: [PyNeoInstance: A User-Friendly Python Library for Neo4j](https://medium.com/@a-gilmore/pyneoinstance-a-user-friendly-python-library-for-neo4j-dbefa3117bb2).
|
|
1000
|
+
* Graph EDA: [Performing graph EDA with the PyNeoInstance Python Package](https://medium.com/@jose.alvarado-guzman/0c0f6f230aa9)
|
|
@@ -0,0 +1,308 @@
|
|
|
1
|
+
# PyNeoInstance Package
|
|
2
|
+
|
|
3
|
+
## What is it?
|
|
4
|
+
|
|
5
|
+
PyNeoInstance is a Python package that provides a user friendly API for submitting Cypher queries to Neo4j and performing Exploratory Data Analysis (EDA) of your graph.
|
|
6
|
+
|
|
7
|
+
It supports the following features:
|
|
8
|
+
|
|
9
|
+
- __<u>Read query</u>__: Submit a Cypher read query, with or without parameters, and get a Pandas Data Frame with the results.
|
|
10
|
+
- __<u>Write query</u>__: Submit a write Cypher query, with or without parameters, to update an existing database.
|
|
11
|
+
- __<u>Write queries</u>__: Submit a list of Cypher queries to update an existing database.
|
|
12
|
+
- __<u>Write query with data</u>__: Update a database based on a Pandas Data Frame and a Cypher query.
|
|
13
|
+
- __<u>Write queries with data</u>__: Update a database based on a Pandas Data Frame and a list of Cypher queries.
|
|
14
|
+
- __<u>Node label frequency</u>__: Get the distribution of the graph node labels.
|
|
15
|
+
- __<u>Multi label node frequency</u>__: Get the distribution of the graph multi label nodes.
|
|
16
|
+
- __<u>Relationship type frequency</u>__: Get the distribution of the graph relationship types.
|
|
17
|
+
- __<u>Relationship type source-target frequency</u>__: Get the distribution of relationship source and target.
|
|
18
|
+
- __<u>Node and relationship properties</u>__: Get information of all properties in the graph.
|
|
19
|
+
- __<u>Constraints</u>__: Get information of all constraints in the graph.
|
|
20
|
+
- __<u>Indexes</u>__: Get information of all indexes in the graph.
|
|
21
|
+
- __<u>Schema Visualization</u>__: Visualize the graph schema.
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
## How to install it?
|
|
25
|
+
|
|
26
|
+
```python
|
|
27
|
+
pip install pyneoinstance
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## How to use it?
|
|
31
|
+
|
|
32
|
+
### Instantiating the Neo4jInstance class
|
|
33
|
+
|
|
34
|
+
In order to use this package, you first need to import the Neo4jInstance class.
|
|
35
|
+
|
|
36
|
+
```python
|
|
37
|
+
from pyneoinstance import Neo4jInstance
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Once imported, you need to instantiate this class by providing the following arguments:
|
|
41
|
+
|
|
42
|
+
- _uri_: The Neo4j instance unique resource identifier.
|
|
43
|
+
- _user_: The user name used to authenticate.
|
|
44
|
+
- _password_: The user name password.
|
|
45
|
+
|
|
46
|
+
```python
|
|
47
|
+
uri = 'bolt://localhost:7687'
|
|
48
|
+
user = 'neo4j'
|
|
49
|
+
password = 'password'
|
|
50
|
+
|
|
51
|
+
graph = Neo4jInstance(uri, user, password)
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Although the previous code is valid, it is not recommended to hard code your user credentials in your application. As an alternative you can store this information as part of your OS user environment variables and then retrieve them in Python.
|
|
55
|
+
|
|
56
|
+
```python
|
|
57
|
+
import os
|
|
58
|
+
|
|
59
|
+
uri = os.environ['NEO4J_URI']
|
|
60
|
+
user = os.environ['NEO4J_USER']
|
|
61
|
+
password = os.environ['NEO4J_PASSWORD']
|
|
62
|
+
graph = Neo4jInstance(uri,user,password)
|
|
63
|
+
```
|
|
64
|
+
Another alternative is to create a YAML configuration file to store this information.
|
|
65
|
+
|
|
66
|
+
```yaml
|
|
67
|
+
# configuration.yaml
|
|
68
|
+
db_info:
|
|
69
|
+
uri: neo4j_uri
|
|
70
|
+
user: neo4j_user
|
|
71
|
+
password: user_password
|
|
72
|
+
database: neo4j
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
This YAML file can then be loaded in Python and this information can be retrieve.
|
|
76
|
+
|
|
77
|
+
```python
|
|
78
|
+
from pyneoinstance import load_yaml_file
|
|
79
|
+
|
|
80
|
+
config = load_yaml_file(configuration.yaml)
|
|
81
|
+
db_info = config['db_info']
|
|
82
|
+
graph = Neo4jInstance(db_info['uri'], db_info['user'], db_info['password'])
|
|
83
|
+
```
|
|
84
|
+
### Submitting a Cypher read query
|
|
85
|
+
|
|
86
|
+
Once you instantiated the Neo4jInstance class, you can submit a Cypher read query by using the class method execute\_read\_query. This method receives the Cypher query string as an argument and optionally the name of the database on which the query should be executed. If the database name is not provided the query will be execute using your default Neo4j database. The results of the query are return as a Pandas Data Frame. To avoid unwanted column names in the resulting Data Frame, always provide aliases for all returned properties.
|
|
87
|
+
|
|
88
|
+
```python
|
|
89
|
+
query = """
|
|
90
|
+
MATCH (m:Movie {title: "The Matrix"})
|
|
91
|
+
RETURN m.tile AS title,
|
|
92
|
+
m.released AS releasedYear,
|
|
93
|
+
m.tagline AS tagline;
|
|
94
|
+
"""
|
|
95
|
+
|
|
96
|
+
data = graph.execute_read_query(query,db_info['database'])
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
### Submitting a Cypher read query with parameters
|
|
100
|
+
|
|
101
|
+
The method execute\_read\_query can optionally received a Python Dictionary with the parameters required as part of you Cypher query.
|
|
102
|
+
|
|
103
|
+
```python
|
|
104
|
+
query = """
|
|
105
|
+
MATCH (m:Movie {title: $movie_title})
|
|
106
|
+
RETURN m.tile AS title,
|
|
107
|
+
m.released AS releasedYear,
|
|
108
|
+
m.tagline AS tagline;
|
|
109
|
+
"""
|
|
110
|
+
|
|
111
|
+
data = graph.execute_read_query(query, db_info['database'], db_info['database'], parameters={'movie_title':'The Matrix'})
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
### Submitting a Cypher write query
|
|
115
|
+
|
|
116
|
+
You can submit a Cypher write query using the method execute_write_query. The signature of this method is identical to the execute_read_query. This method returns a dictionary containing the updates made to the database. As with read queries you can also specify a Python dictionary containing Cypher parameters.
|
|
117
|
+
|
|
118
|
+
```python
|
|
119
|
+
query = """
|
|
120
|
+
MERGE(m:Movie {title:$movie_title})
|
|
121
|
+
ON CREATE SET
|
|
122
|
+
m.release = $year,
|
|
123
|
+
m.tagline = $tagline
|
|
124
|
+
"""
|
|
125
|
+
|
|
126
|
+
results = graph.execute_write_query(query, db_info['database'],
|
|
127
|
+
parameters={
|
|
128
|
+
'movie_title':'Wakanda Forever',
|
|
129
|
+
'year'=2022,
|
|
130
|
+
tagline=''Wakanda Forever! Long Live Wakanda! For Honor, For Legacy, For Wakanda!''
|
|
131
|
+
})
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
### Updating a database with data in a DataFrame: Sequentially
|
|
135
|
+
|
|
136
|
+
We can update a Neo4j database with data contained in a Pandas Data Frame using the methods execute\_write\_query\_with\_data or execute\_write\_queries\_with\_data. These methods return a dictionary containing the numbers and types of updates made to the database. Besides taking the query or list of queries as an argument, we can pass to these methods the following arguments:
|
|
137
|
+
|
|
138
|
+
- _data_: Pandas Data Frame containing data to process.
|
|
139
|
+
- _database_: Optional, database on which to execute the queries.
|
|
140
|
+
- _partitions_: Optional, number of partitions use to break the Data Frame in smaller chucks.
|
|
141
|
+
- _parallel_: Whether to execute the query using multiple Python process.
|
|
142
|
+
- _workers_: The number of processes to spawn to execute the queries in parallel.
|
|
143
|
+
- _parameters_: Extra arguments containing optional cypher parameters.
|
|
144
|
+
|
|
145
|
+
To illustrates how to use these methods lets imagine we have a csv file that looks like this:
|
|
146
|
+
|
|
147
|
+
```python
|
|
148
|
+
# people.csv
|
|
149
|
+
personId,name,birthYear,deathYear
|
|
150
|
+
23945,Gérard Pirès,1942,
|
|
151
|
+
553509,Helen Reddy,1941,
|
|
152
|
+
113934,Susan Flannery,1939,
|
|
153
|
+
26706,David Rintoul,1948,
|
|
154
|
+
237537,Rita Marley,1946,
|
|
155
|
+
11502,Harry Davenport,1866,1949
|
|
156
|
+
11792,George Dickerson,1933,
|
|
157
|
+
7242,Temuera Morrison,1960,
|
|
158
|
+
3482,Chus Lampreave,1930,
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
We want to update our database by loading this data as a node with the label Person containing the properties id, name, birthYear and deathYear. To accomplish this, we need to performed the following steps:
|
|
162
|
+
|
|
163
|
+
1. Load the csv to Python as a Data Frame.
|
|
164
|
+
2. Develop the Cypher query that is going to be use to load the Data Frame into Neo4j.
|
|
165
|
+
3. Call the execute\_write\_query\_with\_data of an existing instance of the Neo4jInstance class passing the Data Frame, Cypher query and any other optional arguments to this method.
|
|
166
|
+
|
|
167
|
+
```python
|
|
168
|
+
import pandas as pd
|
|
169
|
+
|
|
170
|
+
person_data = pd.read_csv('people.csv')
|
|
171
|
+
|
|
172
|
+
person_load_query = """
|
|
173
|
+
WITH $rows AS rows
|
|
174
|
+
UNWIND rows AS row
|
|
175
|
+
MERGE(p:Person {id:row.personId})
|
|
176
|
+
ON CREATE
|
|
177
|
+
SET p.id = row.personId,
|
|
178
|
+
p.name = row.name,
|
|
179
|
+
p.birthYear=toInteger(row.birthYear),
|
|
180
|
+
p.deathYear = toInteger(row.deathYear)
|
|
181
|
+
"""
|
|
182
|
+
|
|
183
|
+
person_load_result = graph.execute_write_query_with_data(person_load_query, person_data, db_info['database'])
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
It is not a good practice to store your Cypher queries as strings within your Python modules. To solve this, we can extend our YAML configuration file to store all our Cypher queries.
|
|
187
|
+
|
|
188
|
+
```yaml
|
|
189
|
+
# configuration.yaml
|
|
190
|
+
db_info:
|
|
191
|
+
uri: neo4j_uri
|
|
192
|
+
user: neo4j_user
|
|
193
|
+
password: user_password
|
|
194
|
+
database: neo4j
|
|
195
|
+
loading_queries:
|
|
196
|
+
nodes:
|
|
197
|
+
Person: |
|
|
198
|
+
WITH $rows AS rows
|
|
199
|
+
UNWIND rows AS row
|
|
200
|
+
MERGE(p:Person {id:row.personId})
|
|
201
|
+
ON CREATE
|
|
202
|
+
SET p.id = row.personId,
|
|
203
|
+
p.name = row.name,
|
|
204
|
+
p.birthYear=toInteger(row.birthYear),
|
|
205
|
+
p.deathYear = toInteger(row.deathYear)
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
Now that we added the loading query to our configuration file, lets re-write our Python script to get this query from the configuration file.
|
|
209
|
+
|
|
210
|
+
```python
|
|
211
|
+
import pandas as pd
|
|
212
|
+
|
|
213
|
+
person_data = pd.read_csv('people.csv')
|
|
214
|
+
|
|
215
|
+
node_load_queries = config['loading_queries']['nodes']
|
|
216
|
+
|
|
217
|
+
person_load_result = graph.execute_write_query_with_data(node_load_queries['Person'], person_data, db_info['database'])
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
If the number of rows in your Data Frame is 100K or more it is recommended to break the Data Frame in smaller chunks so that they can be process in different transactions. This will avoid running out of heap error. To do this, just indicate in the _partitions_ argument how many partitions would you like to break this Data Frame into. These partitions will be loaded in different sequential transactions unless you specified loading them in parallel by setting the _parallel_ argument to true. We will discuss this process in the next section.
|
|
221
|
+
|
|
222
|
+
```python
|
|
223
|
+
person_load_result = graph.execute_write_query_with_data(
|
|
224
|
+
node_load_queries['person'],
|
|
225
|
+
person_data,
|
|
226
|
+
db_info['database'],
|
|
227
|
+
partitions = 5
|
|
228
|
+
)
|
|
229
|
+
```
|
|
230
|
+
## Updating a database with data in a Data Frame: Parallel
|
|
231
|
+
|
|
232
|
+
To load the data in the Data Frame in parallel, we need to partition the Data Frame by incrementing the _partitions_ argument and set the _parallel_ argument to true. By default, this will use all the available virtual CPUs in the machine. If you need to use less, you can specify how many to use by setting the _workers_ argument.
|
|
233
|
+
|
|
234
|
+
```python
|
|
235
|
+
person_load_result = graph.execute_write_query_with_data(
|
|
236
|
+
node_load_queries['person'],
|
|
237
|
+
person_data,
|
|
238
|
+
partitions = 5,
|
|
239
|
+
parallel = True,
|
|
240
|
+
workers = 5
|
|
241
|
+
)
|
|
242
|
+
```
|
|
243
|
+
## Graph EDA
|
|
244
|
+
|
|
245
|
+
### Node Label Frequency
|
|
246
|
+
The get_node_label_freq method provides the frequency and relative frequency of all nodes in the graph. This method consider all node labels as single labels, if you want the distribution considering multiple labels use the get_node_multilabel_freq method.
|
|
247
|
+
```python
|
|
248
|
+
graph.get_node_label_freq(database='mydbname')
|
|
249
|
+
```
|
|
250
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/nodefreq.png" width="250">
|
|
251
|
+
|
|
252
|
+
### Relationship Type Frequency
|
|
253
|
+
The get_rela_type_freq method provides the frequency and relative frequency of all relationship types in the graph.
|
|
254
|
+
```python
|
|
255
|
+
graph.get_rela_type_freq(database='mydbname')
|
|
256
|
+
```
|
|
257
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/relaFreq.png" width="300">
|
|
258
|
+
|
|
259
|
+
### Relationship Source-Target Frequency
|
|
260
|
+
The get_rela_source_target_freq method provides the frequency and relative frequency of all relationship types considering the node label of the source and targets of the relationships.
|
|
261
|
+
```python
|
|
262
|
+
graph.get_rela_type_freq(database='mydbname')
|
|
263
|
+
```
|
|
264
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/relaSourceTargetFreq.png" width="300">
|
|
265
|
+
|
|
266
|
+
### Properties
|
|
267
|
+
The get_properties method provides information about all the nodes and relationships properties.
|
|
268
|
+
```python
|
|
269
|
+
graph.graph.get_properties(database='mydbname')
|
|
270
|
+
```
|
|
271
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/properties.png" width="350" hight="300">
|
|
272
|
+
|
|
273
|
+
### Constraints
|
|
274
|
+
The get_constraints method provides information about all the graph constraints.
|
|
275
|
+
```python
|
|
276
|
+
graph.get_constraints(database='mydbname')
|
|
277
|
+
```
|
|
278
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/constraints.png" width="350" hight="300">
|
|
279
|
+
|
|
280
|
+
### Indexes
|
|
281
|
+
The get_indexes method provides information about all the graph indexes.
|
|
282
|
+
```python
|
|
283
|
+
graph.get_indexes(database='mydbname')
|
|
284
|
+
```
|
|
285
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/indexes.png" width="450" hight="400">
|
|
286
|
+
|
|
287
|
+
### Schema Visualization
|
|
288
|
+
The graph.get_schema_visualization provides an interactive UI with the visualization of the graph schema with the ability to filter the schema. This UI can be access via a Notebook or HTML page. This visualization displays the node labels as string close to nodes, relationship types are only display on mouse hover.
|
|
289
|
+
|
|
290
|
+
```python
|
|
291
|
+
schema = graph.get_schema_visualization(database='mydbname')
|
|
292
|
+
schema.show('schema.html')
|
|
293
|
+
```
|
|
294
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/schema.png" width="450" hight="400">
|
|
295
|
+
|
|
296
|
+
To access the UI via HTTML page just set the parameter notebook to false in the show method and open the html file in your browser.
|
|
297
|
+
|
|
298
|
+
```python
|
|
299
|
+
schema = graph.get_schema_visualization(database='mydbname')
|
|
300
|
+
schema.show('schema.html',notebook=False)
|
|
301
|
+
```
|
|
302
|
+
<img src="https://s3.amazonaws.com/neo4j-ps-ds-bootcamp/img/schemaHtml.png" width="450" hight="400">
|
|
303
|
+
|
|
304
|
+
## More detailed examples
|
|
305
|
+
|
|
306
|
+
For more detailed examples please refer to the following articles:
|
|
307
|
+
* Data ingestion: [PyNeoInstance: A User-Friendly Python Library for Neo4j](https://medium.com/@a-gilmore/pyneoinstance-a-user-friendly-python-library-for-neo4j-dbefa3117bb2).
|
|
308
|
+
* Graph EDA: [Performing graph EDA with the PyNeoInstance Python Package](https://medium.com/@jose.alvarado-guzman/0c0f6f230aa9)
|