grnsight 6.0.4 → 6.0.7

package/database/README.md

@@ -2,10 +2,18 @@
 Here are the files pertaining to both the network and expression databases. Look within the README.md files of both folders for information pertinent to the schema that you intend to be using.
 ## Setting up a local postgres GRNsight Database
 1. Installing PostgreSQL on your computer
-    - MacOS and Windows can follow these [instructions](https://dondi.lmu.build/share/db/postgresql-setup-day.pdf) on how to install postgreSQL.
-        - Step 1 tells you how to install postgreSQL on your local machine, initialize a database, and how to start and stop running your database instance.
-        - If your terminal emits a message that looks like `initdb --locale=C -E UTF-8 location-of-cluster` from Step 1B, then your installer has initialized a database for you.
-        - Additionally, your installer may start the server for you upon installation. To start the server yourself run `pg_ctl start -D location-of-cluster`. To stop the server run `pg_ctl stop -D location-of-cluster`.
+    - MacOS and Windows can follow these instructions on how to install postgreSQL.
+        - Install the software at this [link](https://www.postgresql.org/download/) 
+        - Initialize the database
+           - If your terminal emits a message that looks like `initdb --locale=C -E UTF-8 location-of-cluster` from Step 1B, then your installer has initialized a database for you.
+           - Open the terminal and type the command `initdb --locale=C -E UTF-8 location-of-cluster`
+           - "Cluster" is the PostgreSQL term for the file structure of a PostgreSQL database instance
+           - You will have to modify location-of-cluster to the folder name you want to store the database (you don't need to create a folder, the command will create the folder for you, just create the name)
+        - Start and stop the server
+            - Additionally, your installer may start the server for you upon installation (You can save this command for further reuse).
+            - To start the server yourself run `pg_ctl start -D location-of-cluster` (You can save this command for further reuse).
+            - To stop the server run `pg_ctl stop -D location-of-cluster`.
+        
     - Linux users
       - The MacOS and Windows instructions will _probably_ not work for you. You can try at your own risk to check.
       - Linux users can try these [instructions](https://www.geeksforgeeks.org/install-postgresql-on-linux/) and that should work for you (...maybe...). If it doesn't try googling instructions with your specific operating system. Sorry!
@@ -20,11 +28,11 @@ Here are the files pertaining to both the network and expression databases. Look
            From there, create the schemas using the following commands:
             
             ```
-            CREATE SCHEMA spring2022_network;
+            CREATE SCHEMA gene_regulatory_network;
             ```
             
             ```
-            CREATE SCHEMA fall2021;
+            CREATE SCHEMA gene_expression;
             ```
             
            Once they are created you can exit your database using the command `\q`.
@@ -47,18 +55,32 @@ Here are the files pertaining to both the network and expression databases. Look
           pip3 install pandas requests intermine tzlocal
           ```
           
-          Once the dependencies have been installed, you can run 
-          
+          Once the dependencies have been installed, you can run
           ```
-          python3 <path to GRNsight/database/network-database/scripts>/generate_network.py
+          cd <path to GRNsight/database/network-database/scripts>
+          python3 generate_network.py
           ```
-        
           This will take a while to get all of the network data and generate all of the files. This will create a folder full of the processed files in `database/network-database/script-results`.
+          
+          *** Note: *** If you get an error similar to the following image where it references the in then you are one of the unlucky few who has to edit the intermine.py file directly.
+          
+        ![image](https://user-images.githubusercontent.com/21343072/213089777-dfe772bc-deca-4df7-816f-72703db24d1e.png)
+
+          - Navigate the referenced file ( \<path specific to your machine>/intermine/webservice.py )
+          
+          - The try-catch block should look like this:
+          
+              - ![image](https://user-images.githubusercontent.com/21343072/213094796-c48f54da-b76c-4266-81fb-6aaef24a36c9.png)
+              
+          - Change it to the following, rerun the `generate_network.py` command and it should work! If it doesn't you may need to troubleshoot a bit further (´◕ ᵔ ◕`✿)*ᶜʳᶦᵉˢ*.
+          
+              - ![image](https://user-images.githubusercontent.com/21343072/213094984-bff2deb3-d26b-4809-83d6-6a6615b6e3cf.png)
   
-        2. Load the processed files into your database. 
+        2. Load the processed files into your database.
         
             ```
-            python3 <path to GRNsight/database/network-database/scripts>/loader.py | psql postgresql://localhost/postgres
+            cd <path to GRNsight/database/network-database/scripts>
+            python3 loader.py | psql postgresql://localhost/postgres
             ```
             
             This should output a bunch of COPY print statements to your terminal. Once complete your database is now loaded with the network data.
@@ -70,18 +92,20 @@ Here are the files pertaining to both the network and expression databases. Look
             mkdir <path to GRNsight/database/expression-database>/source-files
             ```
         
-        2. Download the _"Expression 2020"_ folder from Box located in `GRNsight > GRNsight Expression > Expression 2020` to your newly created `source-files` folder
+        2. Download the _"Expression 2020"_ folder from Box located in `GRNsight > GRNsight Expression > Expression 2020` to your newly created `source-files` folder. Your the path should look like this: GRNsight > database > expression-database > source-files > Expression 2020 > [the actual csv and xlsx files are here!]
         3. Run the pre-processing script on the data. This will create a folder full of the processed files in `database/expression-database/script-results`.
         
             ```
-            python3 <path to GRNsight/database/expression-database/scripts>/preprocessing.py
+            cd <path to GRNsight/database/expression-database/scripts>
+            python3 preprocessing.py
             ```
             
         4. Load the processed files into your database. 
         
             ```
-            python3 <path to GRNsight/database/expression-database/scripts>/loader.py | psql postgresql://localhost/postgres
+            cd <path to GRNsight/database/expression-database/scripts>
+            python3 loader.py | psql postgresql://localhost/postgres
             ```
             
             This should output a bunch of COPY print statements to your terminal. Once complete your database is now loaded with the expression data.
-        
+3. Continue setting up in the [Initial Setup Wiki page](https://github.com/dondi/GRNsight/wiki/Initial-Setup)

package/database/expression-database/README.md

@@ -6,9 +6,9 @@ All files pertaining the expression database live within this directory.
 
 #### Schema
 
-All network data is stored within the fall2021 schema on our Postgres database.
+All network data is stored within the gene_expression schema on our Postgres database.
 
-The schema is located within this directory at the top level in the file `schema.sql`. It defines the tables located within the fall2021 schema. 
+The schema is located within this directory at the top level in the file `schema.sql`. It defines the tables located within the gene_expression schema. 
 
 Usage:
 To load to local database

package/database/expression-database/schema.sql

@@ -1,4 +1,4 @@
-CREATE TABLE fall2021.ref (
+CREATE TABLE gene_expression.ref (
   pubmed_id VARCHAR,
   authors VARCHAR,
   publication_year VARCHAR,
@@ -8,7 +8,7 @@ CREATE TABLE fall2021.ref (
   PRIMARY KEY(ncbi_geo_id, pubmed_id)
 );
 
-CREATE TABLE fall2021.gene (
+CREATE TABLE gene_expression.gene (
   gene_id VARCHAR, -- systematic like name
   display_gene_id VARCHAR, -- standard like name
   species VARCHAR,
@@ -16,10 +16,10 @@ CREATE TABLE fall2021.gene (
   PRIMARY KEY(gene_id, taxon_id)
 ); 
 
-CREATE TABLE fall2021.expression_metadata (
+CREATE TABLE gene_expression.expression_metadata (
   ncbi_geo_id VARCHAR,
   pubmed_id VARCHAR,
-  FOREIGN KEY (ncbi_geo_id, pubmed_id) REFERENCES fall2021.ref(ncbi_geo_id, pubmed_id),
+  FOREIGN KEY (ncbi_geo_id, pubmed_id) REFERENCES gene_expression.ref(ncbi_geo_id, pubmed_id),
   control_yeast_strain VARCHAR,
   treatment_yeast_strain VARCHAR,
   control VARCHAR,
@@ -33,10 +33,10 @@ CREATE TABLE fall2021.expression_metadata (
   display_expression_table VARCHAR,
   PRIMARY KEY(ncbi_geo_id, pubmed_id, time_value)
 );
-CREATE TABLE fall2021.expression (
+CREATE TABLE gene_expression.expression (
   gene_id VARCHAR,
   taxon_id VARCHAR,
-  FOREIGN KEY (gene_id, taxon_id) REFERENCES fall2021.gene(gene_id, taxon_id),
+  FOREIGN KEY (gene_id, taxon_id) REFERENCES gene_expression.gene(gene_id, taxon_id),
   -- ncbi_geo_id VARCHAR,
   -- pubmed_id VARCHAR,
   sort_index INT,
@@ -45,27 +45,27 @@ CREATE TABLE fall2021.expression (
   time_point FLOAT,
   dataset VARCHAR, 
   PRIMARY KEY(gene_id, sample_id)
-  -- FOREIGN KEY (ncbi_geo_id, pubmed_id, time_point) REFERENCES fall2021.expression_metadata(ncbi_geo_id, pubmed_id, time_value)
+  -- FOREIGN KEY (ncbi_geo_id, pubmed_id, time_point) REFERENCES gene_expression.expression_metadata(ncbi_geo_id, pubmed_id, time_value)
 ); 
-CREATE TABLE fall2021.degradation_rate (
+CREATE TABLE gene_expression.degradation_rate (
   gene_id VARCHAR,
   taxon_id VARCHAR,
-  FOREIGN KEY (gene_id, taxon_id) REFERENCES fall2021.gene(gene_id, taxon_id),
+  FOREIGN KEY (gene_id, taxon_id) REFERENCES gene_expression.gene(gene_id, taxon_id),
   ncbi_geo_id VARCHAR,
   pubmed_id VARCHAR,
-  FOREIGN KEY (ncbi_geo_id, pubmed_id) REFERENCES fall2021.ref(ncbi_geo_id, pubmed_id),
+  FOREIGN KEY (ncbi_geo_id, pubmed_id) REFERENCES gene_expression.ref(ncbi_geo_id, pubmed_id),
   PRIMARY KEY(gene_id, ncbi_geo_id, pubmed_id),
   degradation_rate FLOAT
 );
 
-CREATE TABLE fall2021.production_rate (
+CREATE TABLE gene_expression.production_rate (
   gene_id VARCHAR,
   taxon_id VARCHAR,
-  FOREIGN KEY (gene_id, taxon_id) REFERENCES fall2021.gene(gene_id, taxon_id),
+  FOREIGN KEY (gene_id, taxon_id) REFERENCES gene_expression.gene(gene_id, taxon_id),
   ncbi_geo_id VARCHAR,
   pubmed_id VARCHAR,
-  FOREIGN KEY (ncbi_geo_id, pubmed_id) REFERENCES fall2021.ref(ncbi_geo_id, pubmed_id),
+  FOREIGN KEY (ncbi_geo_id, pubmed_id) REFERENCES gene_expression.ref(ncbi_geo_id, pubmed_id),
   PRIMARY KEY(gene_id, ncbi_geo_id, pubmed_id),
   production_rate FLOAT
-  -- FOREIGN KEY (gene_id, ncbi_geo_id, pubmed_id) REFERENCES fall2021.degradation_rate(gene_id, ncbi_geo_id, pubmed_id) -- not sure if we want to link the generated production rate to it's original degradation rate
+  -- FOREIGN KEY (gene_id, ncbi_geo_id, pubmed_id) REFERENCES gene_expression.degradation_rate(gene_id, ncbi_geo_id, pubmed_id) -- not sure if we want to link the generated production rate to it's original degradation rate
 );

package/database/expression-database/scripts/loader.py

@@ -45,7 +45,7 @@ def convert_int(potential_int):
 This program Loads Refs into the database
 """
 def LOAD_REFS():
-    print('COPY fall2021.ref (pubmed_id, authors, publication_year, title, doi, ncbi_geo_id) FROM stdin;')
+    print('COPY gene_expression.ref (pubmed_id, authors, publication_year, title, doi, ncbi_geo_id) FROM stdin;')
     REFS_SOURCE = '../script-results/processed-expression/refs.csv'
     with open(REFS_SOURCE, 'r+') as f:
         reader = csv.reader(f)
@@ -67,7 +67,7 @@ def LOAD_REFS():
 This program Loads ID Mapping into the database
 """
 def LOAD_GENES():
-    print('COPY fall2021.gene (gene_id, display_gene_id, species, taxon_id) FROM stdin;')
+    print('COPY gene_expression.gene (gene_id, display_gene_id, species, taxon_id) FROM stdin;')
     GENE_SOURCE = '../script-results/processed-expression/genes.csv'
     with open(GENE_SOURCE, 'r+') as f:
         reader = csv.reader(f)
@@ -87,7 +87,7 @@ def LOAD_GENES():
 This program Loads Expression Metadata into the database
 """
 def LOAD_EXPRESSION_METADATA():
-    print('COPY fall2021.expression_metadata (ncbi_geo_id, pubmed_id, control_yeast_strain, treatment_yeast_strain, control, treatment, concentration_value, concentration_unit, time_value, time_unit, number_of_replicates, expression_table) FROM stdin;')
+    print('COPY gene_expression.expression_metadata (ncbi_geo_id, pubmed_id, control_yeast_strain, treatment_yeast_strain, control, treatment, concentration_value, concentration_unit, time_value, time_unit, number_of_replicates, expression_table) FROM stdin;')
     EXPRESSION_METADATA_SOURCE = '../script-results/processed-expression/expression-metadata.csv'
     with open(EXPRESSION_METADATA_SOURCE, 'r+') as f:
         reader = csv.reader(f)
@@ -116,7 +116,7 @@ def LOAD_EXPRESSION_METADATA():
 This program Loads Expression Data into the database
 """
 def LOAD_EXPRESSION_DATA():
-    print('COPY fall2021.expression (gene_id, taxon_id, sort_index, sample_id, expression, time_point, dataset) FROM stdin;')
+    print('COPY gene_expression.expression (gene_id, taxon_id, sort_index, sample_id, expression, time_point, dataset) FROM stdin;')
     EXPRESSION_DATA_SOURCE = '../script-results/processed-expression/expression-data.csv'
     with open(EXPRESSION_DATA_SOURCE, 'r+') as f:
         reader = csv.reader(f)
@@ -140,7 +140,7 @@ def LOAD_EXPRESSION_DATA():
 This program Loads Production Rates into the database
 """
 def LOAD_PRODUCTION_RATES():
-    print('COPY fall2021.production_rate (gene_id, taxon_id, ncbi_geo_id, pubmed_id, production_rate) FROM stdin;')
+    print('COPY gene_expression.production_rate (gene_id, taxon_id, ncbi_geo_id, pubmed_id, production_rate) FROM stdin;')
     PRODUCTION_RATES_SOURCE = '../script-results/processed-expression/production-rates.csv'
     with open(PRODUCTION_RATES_SOURCE, 'r+') as f:
         reader = csv.reader(f)
@@ -161,7 +161,7 @@ def LOAD_PRODUCTION_RATES():
 This program Loads Degradation Rates into the database
 """
 def LOAD_DEGRADATION_RATES():
-    print('COPY fall2021.degradation_rate (gene_id, taxon_id, ncbi_geo_id, pubmed_id, degradation_rate) FROM stdin;')
+    print('COPY gene_expression.degradation_rate (gene_id, taxon_id, ncbi_geo_id, pubmed_id, degradation_rate) FROM stdin;')
     DEGRADATION_RATES_SOURCE = '../script-results/processed-expression/degradation-rates.csv'
     with open(DEGRADATION_RATES_SOURCE, 'r+') as f:
         reader = csv.reader(f)

package/database/network-database/README.md

@@ -6,9 +6,9 @@ All files pertaining the network database live within this directory.
 
 ### Schema
 
-All network data is stored within the spring2022_network schema on our Postgres database.
+All network data is stored within the gene_regulatory_network schema on our Postgres database.
 
-The schema is located within this directory at the top level in the file `schema.sql`. It defines the tables located within the spring2022_network schema. 
+The schema is located within this directory at the top level in the file `schema.sql`. It defines the tables located within the gene_regulatory_network schema. 
 
 Usage:
 To load to local database
@@ -32,10 +32,13 @@ Within the scripts directory, there are the following files:
 
 - `generate_network.py`
 - `loader.py`
+- `generate_new_network_verion.py`
+- `loader_updates.py`
 - `filter_genes.py`
 - `generate_sgd_network_from_yeastract_network.py`
 
-#### Network Generator (and data preprocessor)
+
+#### Network Generator (and data preprocessor) (FOR FRESH DATABASE INSTALLS ONLY)
 
 This script (`generate_network.py`) is a two-for-one. It first uses the yeastmine service from the SGD database to query for all regulator genes relating to Saccharomyces cerevisiae. From there it gets all all of the targets for each regulator gene. We then construct two networks from these connections (a regulator by regulator matrix as well as a regulator by target matrix). We also construct the processed loader files, so that they are ready to load using `loader.py`.
 
@@ -47,7 +50,7 @@ Usage:
 ```
 python3 generate_network.py
 ```
-#### Database Loader
+#### Database Loader (FOR FRESH DATABASE INSTALLS ONLY)
 
 This script (`loader.py`) is to be used to load your preprocessed genes into the database. 
 
@@ -62,6 +65,35 @@ To load to production database
 ```
 python3 loader.py | psql <address to database>
 ```
+#### Network Generator (and data preprocessor) (FOR UPDATES TO EXISTING DATABASE ONLY)
+
+This script (`generate_new_network_verion.py`) is similar to its counterpart `generate_network.py`. It gets all existing genes in the database using the environment variable 'DB_URL'. You can set this environment variable on the terminal right before the command.  It uses the yeastmine service from the SGD database to query for all regulator genes relating to Saccharomyces cerevisiae. From there it gets all all of the targets for each regulator gene. We then construct two networks from these connections (a regulator by regulator matrix as well as a regulator by target matrix). We then see if the genes in the newly constructed network have any updates (i.e a gene's standard name was set or a new gene was added to the database). We also construct the processed loader files, so that they are ready to load using `loader_updates.py`.
+
+The resulting network matrices are located in `script-results/networks` and the resulting processed loader files are located within `script-results/processed-loader-files`
+
+Make sure to have all dependencies installed beforehand or you will recieve errors. (pip3 install intermine, tzlocal, etc. [see file for all imports]
+
+Usage: 
+```
+DB_URL="postgresql://[<db_user>:<password>]@<address to database>/<database name>" python3 generate_new_network_version.py
+```
+#### Database Loader (FOR UPDATES TO EXISTING DATABASE ONLY)
+
+This script (`loader_updates.py`) is to be used to load your preprocessed genes into the database. 
+
+This program generates direct SQL statements from the source files generated by the network generator in order to populate a relational database with those files’ data as well as make any needed updates to existing genes within the database. If necessary you will be prompted to enter a password.
+
+Usage: 
+To load to local database
+```
+python3 loader_updates.py | psql postgresql://localhost/postgres
+```
+To load to production database
+```
+python3 loader_updates.py | psql -h <grnsight database link> -U <user> <database name>
+
+```
+
 
 #### Filter Genes (beta functionality, not tested)
 

package/database/network-database/schema.sql

@@ -1,11 +1,11 @@
-CREATE TABLE spring2022_network.source (
+CREATE TABLE gene_regulatory_network.source (
   time_stamp TIMESTAMP WITH TIME ZONE,
   source VARCHAR,
-  source_display_name VARCHAR,
+  display_name VARCHAR,
   PRIMARY KEY(time_stamp, source)
 );
 
-CREATE TABLE spring2022_network.gene (
+CREATE TABLE gene_regulatory_network.gene (
   gene_id VARCHAR, -- systematic like name
   display_gene_id VARCHAR, -- standard like name
   species VARCHAR,
@@ -13,13 +13,13 @@ CREATE TABLE spring2022_network.gene (
   regulator BOOLEAN,
   PRIMARY KEY(gene_id, taxon_id)
 ); 
-CREATE TABLE spring2022_network.network (
+CREATE TABLE gene_regulatory_network.network (
   regulator_gene_id VARCHAR,
   target_gene_id VARCHAR,
   taxon_id VARCHAR,
   time_stamp TIMESTAMP WITH TIME ZONE,
   source VARCHAR,
-  FOREIGN KEY (regulator_gene_id, taxon_id) REFERENCES spring2022_network.gene(gene_id, taxon_id),
-  FOREIGN KEY (target_gene_id, taxon_id) REFERENCES spring2022_network.gene(gene_id, taxon_id),
-  FOREIGN KEY (time_stamp, source) REFERENCES spring2022_network.source(time_stamp, source)
+  FOREIGN KEY (regulator_gene_id, taxon_id) REFERENCES gene_regulatory_network.gene(gene_id, taxon_id),
+  FOREIGN KEY (target_gene_id, taxon_id) REFERENCES gene_regulatory_network.gene(gene_id, taxon_id),
+  FOREIGN KEY (time_stamp, source) REFERENCES gene_regulatory_network.source(time_stamp, source)
 ); 

package/database/network-database/scripts/filter_genes.py

@@ -13,7 +13,7 @@ try:
                                   port="5432",
                                   database="postgres")
     cursor = connection.cursor()
-    postgreSQL_select_Query = "select * from spring2022_network.gene"
+    postgreSQL_select_Query = "select * from gene_regulatory_network.gene"
 
     cursor.execute(postgreSQL_select_Query)
     print("Selecting rows from gene table using cursor.fetchall")

package/database/network-database/scripts/generate_network.py

@@ -140,7 +140,7 @@ regulator_to_regulator_file.close()
 # Source Table
 
 SOURCE_DESTINATION = '../script-results/processed-loader-files/source.csv'
-timestamp = datetime.datetime.now(datetime.timezone.utc)
+timestamp = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0)
 
 source = "YeastMine - Saccharomyces Genome Database"
 display_name = "Yeastmine - SGD"

package/database/network-database/scripts/generate_new_network_version.py

@@ -0,0 +1,223 @@
+from __future__ import print_function
+
+from intermine.webservice import Service
+service = Service("https://yeastmine.yeastgenome.org/yeastmine/service")
+from sqlalchemy import create_engine
+
+import csv
+import re
+import sys
+import os
+import datetime
+
+
+def get_all_genes():
+    db = create_engine(os.environ['DB_URL'])
+    
+    with db.connect() as connection:
+        result_set = connection.execute(
+        f"""
+            SELECT display_gene_id, gene_id FROM gene_regulatory_network.gene;
+        """)
+        result = result_set.fetchall()
+        return list(result)
+
+
+
+# Get Network Data from Yeastmine
+
+query = service.new_query("Gene")
+
+query.add_view(
+    "primaryIdentifier", "secondaryIdentifier", "symbol", "name", "sgdAlias",
+    "regulationSummary.summaryParagraph",
+    "regulationSummary.publications.pubMedId",
+    "regulationSummary.publications.citation"
+)
+query.outerjoin("regulationSummary.publications")
+
+regulators = {}
+all_genes = {}
+print("COLLECTING REGULATORS\n")
+for row in query.rows():
+    systematic_name = row["secondaryIdentifier"]
+    standard_name = row["symbol"]
+    if standard_name == None:
+        standard_name = systematic_name
+    
+    regulators[standard_name] =  systematic_name
+    all_genes[standard_name] = systematic_name
+
+regulators_to_targets = {}
+all_targets = {}
+
+
+print("COLLECTING TARGETS\n")
+for regulator in regulators:
+    query = service.new_query("Gene")
+    query.add_constraint("regulatoryRegions", "TFBindingSite")
+    query.add_view(
+        "regulatoryRegions.regulator.symbol",
+        "regulatoryRegions.regulator.secondaryIdentifier", "symbol",
+        "secondaryIdentifier", "regulatoryRegions.regEvidence.ontologyTerm.name",
+        "regulatoryRegions.regEvidence.ontologyTerm.identifier",
+        "regulatoryRegions.experimentCondition",
+        "regulatoryRegions.strainBackground",
+        "regulatoryRegions.regulationDirection",
+        "regulatoryRegions.publications.pubMedId", "regulatoryRegions.datasource",
+        "regulatoryRegions.annotationType"
+    )
+    query.add_sort_order("Gene.secondaryIdentifier", "ASC")
+    query.add_constraint("regulatoryRegions.regulator", "LOOKUP", regulator, "S. cerevisiae", code="A")
+    targets = {}
+
+    for row in query.rows():
+        target_systematic_name = row["secondaryIdentifier"]
+        target_standard_name = row["symbol"]
+        if target_standard_name == None:
+            target_standard_name = target_systematic_name
+        targets[target_standard_name] = target_systematic_name
+        all_targets[target_standard_name] = target_systematic_name
+        all_genes[target_standard_name] =  target_systematic_name
+
+    regulators_to_targets[regulator] = { "systematic_name": regulators[regulator], "targets": targets}
+
+
+
+def create_regulator_to_target_row(target, all_regulators):
+    result = "" + target
+    for regulator in all_regulators:
+        if target in all_regulators[regulator]["targets"]:
+            result += "\t" + "1"
+        else: 
+            result += "\t" + "0"
+    return result
+
+
+# Create files
+
+# Create folder paths 
+if not os.path.exists('../script-results'):
+    os.makedirs('../script-results')
+
+if not os.path.exists('../script-results/networks'):
+    os.makedirs('../script-results/networks')
+
+if not os.path.exists('../script-results/processed-loader-files'):
+    os.makedirs('../script-results/processed-loader-files')
+
+
+
+# Files to be generated
+
+# Generate Networks
+
+REGULATORS_TO_TARGETS_MATRIX = '../script-results/networks/regulators_to_targets.csv'
+REGULATORS_TO_REGULATORS_MATRIX = '../script-results/networks/regulators_to_regulators.csv'
+
+
+targets = []
+for target in all_targets:
+    if target != None:
+        targets.append(target)
+
+regulators_list = []
+for regulator in regulators_to_targets:
+    if regulator != None:
+        regulators_list.append(regulator)
+
+print(f'Creating REGULATORS TO TARGETS MATRIX\n')
+regulator_to_target_file = open(REGULATORS_TO_TARGETS_MATRIX, 'w')
+headers = "cols regulators/rows targets"
+headers += '\t'.join(regulators_list)
+regulator_to_target_file.write(f'{headers}\n')
+for target in targets:
+  result = create_regulator_to_target_row(target, regulators_to_targets)
+  if result != False:
+    regulator_to_target_file.write(f'{result}\n')
+regulator_to_target_file.close()
+
+print(f'Creating REGULATORS TO TARGETS MATRIX\n')
+regulator_to_regulator_file = open(REGULATORS_TO_REGULATORS_MATRIX, 'w')
+headers = "cols regulators/rows targets"
+headers += '\t'.join(regulators_list)
+regulator_to_regulator_file.write(f'{headers}\n')
+for target in targets:
+  result = create_regulator_to_target_row(target, regulators_to_targets)
+  if result != False:
+    regulator_to_regulator_file.write(f'{result}\n')
+regulator_to_regulator_file.close()
+
+
+
+# Create loader-files
+
+# Source Table
+
+SOURCE_DESTINATION = '../script-results/processed-loader-files/source.csv'
+timestamp = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0)
+
+source = "YeastMine - Saccharomyces Genome Database"
+display_name = "Yeastmine - SGD"
+
+source_file = open(SOURCE_DESTINATION, 'w')
+headers = f'Timestamp\tSource\tDisplay Name\n{timestamp}\t{source}\t{display_name}'
+source_file.write(f'{headers}\n')
+source_file.close()
+
+# Gene Table
+
+GENE_DESTINATION = '../script-results/processed-loader-files/gene.csv'
+
+database_genes = {g[1] : g[0] for g in get_all_genes()}
+database_standard_names = [g[0] for g in get_all_genes()]
+database_systematic_names = [g[1] for g in get_all_genes()]
+species = "Saccharomyces cerevisiae"
+taxon_id = "559292"
+genes_to_update = {}
+print(f'Creating gene.csv\n')
+gene_file = open(GENE_DESTINATION, 'w')
+headers = f'Gene ID\tDisplay Gene ID\tSpecies\tTaxon ID\tRegulator'
+gene_file.write(f'{headers}\n')
+for gene in all_genes:
+    if gene not in database_standard_names:
+        if all_genes[gene] not in database_systematic_names:
+            if gene in regulators:
+                gene_file.write(f'{all_genes[gene]}\t{gene}\t{species}\t{taxon_id}\ttrue\n')
+            else:
+                gene_file.write(f'{all_genes[gene]}\t{gene}\t{species}\t{taxon_id}\tfalse\n')
+        else:
+            # if systematic name is found, then we need to update the gene
+            genes_to_update[gene] = all_genes[gene]
+gene_file.close()
+
+# Gene Table Updates
+
+GENE_UPDATES_DESTINATION = '../script-results/processed-loader-files/gene_update.csv'
+
+
+print(f'Creating gene_update.csv\n')
+gene_file = open(GENE_UPDATES_DESTINATION, 'w')
+headers = f'Gene ID\tDisplay Gene ID\tRegulator'
+gene_file.write(f'{headers}\n')
+for gene in genes_to_update:
+    if gene in regulators:
+        gene_file.write(f'{all_genes[gene]}\t{gene}\ttrue\n')
+    else:
+        gene_file.write(f'{all_genes[gene]}\t{gene}\tfalse\n')
+gene_file.close()
+
+
+# Network Table
+
+NETWORK_DESTINATION = '../script-results/processed-loader-files/network.csv'
+
+
+print(f'Creating network.csv\n')
+network_file = open(NETWORK_DESTINATION, 'w')
+headers = f'Regulator Gene ID\tTarget Gene ID\tTaxon ID\tTimestamp\tSource'
+network_file.write(f'{headers}\n')
+for gene in regulators_to_targets:
+    for target_gene in regulators_to_targets[gene]["targets"]:
+        network_file.write(f'{regulators_to_targets[gene]["systematic_name"]}\t{regulators_to_targets[gene]["targets"][target_gene]}\t{taxon_id}\t{timestamp}\t{source}\n')
+network_file.close()

package/database/network-database/scripts/loader.py

@@ -15,7 +15,7 @@ directly into a database command line utility such as `psql`.
 This function Loads Network Data Sources into the database
 """
 def LOAD_SOURCES():
-    print('COPY spring2022_network.source (time_stamp, source, display_name) FROM stdin;')
+    print('COPY gene_regulatory_network.source (time_stamp, source, display_name) FROM stdin;')
     NETWORK_DATA_SOURCE = '../script-results/processed-loader-files/source.csv'
     with open(NETWORK_DATA_SOURCE, 'r+') as f:
         reader = csv.reader(f)
@@ -35,7 +35,7 @@ def LOAD_SOURCES():
 This function Loads Gene ID Mapping into the database
 """
 def LOAD_GENES():
-    print('COPY spring2022_network.gene (gene_id, display_gene_id, species, taxon_id, regulator) FROM stdin;')
+    print('COPY gene_regulatory_network.gene (gene_id, display_gene_id, species, taxon_id, regulator) FROM stdin;')
     GENE_SOURCE = '../script-results/processed-loader-files/gene.csv'
     with open(GENE_SOURCE, 'r+') as f:
         reader = csv.reader(f)
@@ -57,7 +57,7 @@ def LOAD_GENES():
 This function Loads the Network Matrix into the database
 """
 def LOAD_NETWORK():
-    print('COPY spring2022_network.network (regulator_gene_id, target_gene_id, taxon_id, time_stamp, source) FROM stdin;')
+    print('COPY gene_regulatory_network.network (regulator_gene_id, target_gene_id, taxon_id, time_stamp, source) FROM stdin;')
     NETWORK_SOURCE = '../script-results/processed-loader-files/network.csv'
     with open(NETWORK_SOURCE, 'r+') as f:
         reader = csv.reader(f)