citation-map 2.0__tar.gz → 3.1__tar.gz

{citation-map-2.0 → citation-map-3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: citation-map
-Version: 2.0
+Version: 3.1
 Summary: A simple tool to generate your Google Scholar citation world map.
 Home-page: https://github.com/ChenLiu-1996/CitationMap
 Author: Chen Liu
@@ -39,8 +39,6 @@ If you open it on a browser, you will see your own version of the following cita
 
 <img src = "assets/citation_world_map.png" width=800>
 
-## News
-`citation-map` has been upgraded to version 2.0! I finally managed to drastically speed up the process using multiprocessing (without being blocked by Google Scholar). On my personal computer, processing my profile with 100 citations took 1 hour with version 1.0 while it's now taking 5 minutes with version 2.0.
 
 ## Usage
 1. Install the package.
@@ -88,6 +86,27 @@ If you open it on a browser, you will see your own version of the following cita
         If true, print the list of citing institutions (affiliations of citing authors).
     ```
 
+## Changelog
+
+### Version 3.0
+I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map. For example, the string "Yale University" is a much better entry than "Assistant Professor at Yale University".
+
+I applied a simple fix with some rule-based language processing. This helps us identify many missing citing locations.
+
+### Version 2.0
+I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
+
+On my personal computer, processing my profile with 100 citations took 1 hour with version 1.0 while it's now taking 5 minutes with version 2.0.
+
+With that said, please be careful and do not run this tool frequently. I can easily get on Google Scholar's blacklist after a few runs.
+
+### Version 1.0
+Very basic functionality.
+
+This script is a bit slow. On my personal computer, it takes half a minute to process each citation. If you have thousands of citations, it may or may not be a good idea to use this script.
+
+I tried to use multiprocessing, but unfortunately the excessive visits get me blocked by Google Scholar.
+
 ## Dependencies
 Dependencies (`scholarly`, `geopy`, `folium`, `tqdm`) are already taken care of when you install via pip.
 

{citation-map-2.0 → citation-map-3.1}/README.md

@@ -23,8 +23,6 @@ If you open it on a browser, you will see your own version of the following cita
 
 <img src = "assets/citation_world_map.png" width=800>
 
-## News
-`citation-map` has been upgraded to version 2.0! I finally managed to drastically speed up the process using multiprocessing (without being blocked by Google Scholar). On my personal computer, processing my profile with 100 citations took 1 hour with version 1.0 while it's now taking 5 minutes with version 2.0.
 
 ## Usage
 1. Install the package.
@@ -72,6 +70,27 @@ If you open it on a browser, you will see your own version of the following cita
         If true, print the list of citing institutions (affiliations of citing authors).
     ```
 
+## Changelog
+
+### Version 3.0
+I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map. For example, the string "Yale University" is a much better entry than "Assistant Professor at Yale University".
+
+I applied a simple fix with some rule-based language processing. This helps us identify many missing citing locations.
+
+### Version 2.0
+I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
+
+On my personal computer, processing my profile with 100 citations took 1 hour with version 1.0 while it's now taking 5 minutes with version 2.0.
+
+With that said, please be careful and do not run this tool frequently. I can easily get on Google Scholar's blacklist after a few runs.
+
+### Version 1.0
+Very basic functionality.
+
+This script is a bit slow. On my personal computer, it takes half a minute to process each citation. If you have thousands of citations, it may or may not be a good idea to use this script.
+
+I tried to use multiprocessing, but unfortunately the excessive visits get me blocked by Google Scholar.
+
 ## Dependencies
 Dependencies (`scholarly`, `geopy`, `folium`, `tqdm`) are already taken care of when you install via pip.
 

{citation-map-2.0 → citation-map-3.1}/citation_map/citation_map.py

@@ -1,6 +1,9 @@
+import re
 import random
 import time
 import itertools
+
+from typing import List, Tuple
 from scholarly import scholarly, ProxyGenerator
 from geopy.geocoders import Nominatim
 import folium
@@ -8,7 +11,7 @@ from tqdm import tqdm
 from multiprocessing import Pool
 
 
-def fetch_google_scholar_profile(scholar_id):
+def fetch_google_scholar_profile(scholar_id: str):
     '''
     Step 1: Fetch Google Scholar Profile using Scholar ID.
     '''
@@ -16,49 +19,73 @@ def fetch_google_scholar_profile(scholar_id):
     author = scholarly.fill(author, sections=['publications'])
     return author
 
-def find_all_citing_institutions(publications, num_processes: int = 16):
+def find_all_citing_institutions(publications, num_processes: int = 16) -> List[Tuple[str]]:
     '''
     Step 2. Final all citing institutions (i.e., insitutions of citing authors).
     '''
-
     # Fetch metadata for all publications, in parallel.
     with Pool(processes=num_processes) as pool:
-        all_publications = list(tqdm(pool.imap(__fill_publication_metadata, publications), desc='Filling metadata for all publications...', total=len(publications)))
+        all_publications = list(tqdm(pool.imap(__fill_publication_metadata, publications),
+                                     desc='Filling metadata for all publications...',
+                                     total=len(publications)))
 
     # Find all citing papers from all publications, in parallel.
     with Pool(processes=num_processes) as pool:
-        all_citing_papers_nested = list(tqdm(pool.imap(__citing_papers_from_publication, all_publications), desc='Finding all citations...', total=len(all_publications)))
+        all_citing_papers_nested = list(tqdm(pool.imap(__citing_papers_from_publication, all_publications),
+                                             desc='Finding all citations...',
+                                             total=len(all_publications)))
     all_citing_papers = list(itertools.chain(*all_citing_papers_nested))
 
     # Find all citing authors from all citing papers, in parallel.
     with Pool(processes=num_processes) as pool:
-        all_citing_authors_nested = list(tqdm(pool.imap(__authors_from_papers, all_citing_papers), desc='Finding all citing authors...', total=len(all_citing_papers)))
+        all_citing_authors_nested = list(tqdm(pool.imap(__authors_from_papers, all_citing_papers),
+                                              desc='Finding all citing authors...',
+                                              total=len(all_citing_papers)))
     all_citing_authors = list(itertools.chain(*all_citing_authors_nested))
 
     # Find all citing insitutions from all citing authors, in parallel.
     with Pool(processes=num_processes) as pool:
-        all_citing_insitutions = list(tqdm(pool.imap(__institutions_from_authors, all_citing_authors), desc='Finding all citing institutions...', total=len(all_citing_authors)))
+        author_institution_tuple_list = list(tqdm(pool.imap(__institutions_from_authors, all_citing_authors),
+                                                  desc='Finding all citing institutions...',
+                                                  total=len(all_citing_authors)))
 
-    return sorted(all_citing_insitutions)
+    return author_institution_tuple_list
 
-def text_to_geocode(institutions):
+def clean_institution_names(author_institution_tuple_list: List[Tuple[str]]) -> List[Tuple[str]]:
     '''
-    Step 3: Convert institutions in plain text to Geocode.
+    Step 3. Clean up the name of institutions from the author's self-entered Google Scholar affiliation.
     '''
-    geolocator = Nominatim(user_agent='citation_mapper')
-    coordinates = []
-    for institution in institutions:
-        try:
-            geo_location = geolocator.geocode(institution)
-            if geo_location:
-                coordinates.append((geo_location.latitude, geo_location.longitude, institution))
-        except Exception as e:
-            print(f'Error geocoding institution {institution}: {e}')
+    cleaned_author_institution_tuple_list = []
+    for author_name, institution_string in author_institution_tuple_list:
+        # Use a regular expression to split the string by ',', ';', or 'and'.
+        substring_list = [part.strip() for part in re.split(r'[;,]|\band\b', institution_string)]
+        for substring in substring_list:
+            # Use a regular expression to remove anything before 'at', or '@'.
+            cleaned_institution = re.sub(r'.*?\bat\b|.*?@', '', substring, flags=re.IGNORECASE).strip()
+            # Use a regular expression to check if the string is a common weird string.
+            is_common_weird_string = re.search(
+                re.compile(
+                    r'\b(director|manager|chair|engineer|programmer|scientist|professor|lecturer|phd|student|ph\.d|department of)\b',
+                    re.IGNORECASE),
+                substring)
+            if not is_common_weird_string:
+                cleaned_author_institution_tuple_list.append((author_name, cleaned_institution))
+    return cleaned_author_institution_tuple_list
+
+def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]], num_processes: int = 16) -> List[Tuple[str]]:
+    '''
+    Step 4: Convert institutions in plain text to Geocode.
+    '''
+    with Pool(processes=num_processes) as pool:
+        coordinates = list(tqdm(pool.imap(__text_to_geocode, author_institution_tuple_list),
+                                desc='Finding all geographic coordinates...',
+                                total=len(author_institution_tuple_list)))
+    coordinates = [item for item in coordinates if item is not None]  # Filter out empty coordinates.
     return coordinates
 
-def create_map(coordinates, pin_colorful: bool = True):
+def create_map(coordinates: List[Tuple[str]], pin_colorful: bool = True):
     '''
-    Step 4: Create the Citation World Map.
+    Step 5: Create the Citation World Map.
     '''
     citation_map = folium.Map(location=[20, 0], zoom_start=2)
     if pin_colorful:
@@ -66,12 +93,12 @@ def create_map(coordinates, pin_colorful: bool = True):
                   'lightred', 'beige', 'darkblue', 'darkgreen', 'cadetblue',
                   'darkpurple', 'pink', 'lightblue', 'lightgreen',
                   'gray', 'black', 'lightgray']
-        for lat, lon, location in coordinates:
+        for lat, lon, author_name, location in coordinates:
             color = random.choice(colors)
-            folium.Marker([lat, lon], popup=location, icon=folium.Icon(color=color)).add_to(citation_map)
+            folium.Marker([lat, lon], popup='%s (%s)' % (location, author_name), icon=folium.Icon(color=color)).add_to(citation_map)
     else:
-        for lat, lon, location in coordinates:
-            folium.Marker([lat, lon], popup=location).add_to(citation_map)
+        for lat, lon, author_name, location in coordinates:
+            folium.Marker([lat, lon], popup='%s (%s)' % (location, author_name)).add_to(citation_map)
     return citation_map
 
 def __fill_publication_metadata(pub):
@@ -97,9 +124,19 @@ def __institutions_from_authors(author_id):
     citing_author = scholarly.search_author_id(author_id)
     citing_author = scholarly.fill(citing_author)
     if 'affiliation' in citing_author:
-        return citing_author['affiliation']
+        return (citing_author['name'], citing_author['affiliation'])
     return []
 
+def __text_to_geocode(author_institution_tuple):
+    geolocator = Nominatim(user_agent='citation_mapper')
+    try:
+        author_name, institution_name = author_institution_tuple
+        geo_location = geolocator.geocode(institution_name)
+        if geo_location:
+            return (geo_location.latitude, geo_location.longitude, author_name, institution_name)
+    except:
+        return None
+
 def generate_citation_map(scholar_id: str,
                           output_path: str = 'citation_map.html',
                           num_processes: int = 16,
@@ -141,20 +178,27 @@ def generate_citation_map(scholar_id: str,
     author_profile = fetch_google_scholar_profile(scholar_id)
     print('\nAuthor profile found, with %d publications.\n' % len(author_profile['publications']))
 
-    institutions = find_all_citing_institutions(author_profile['publications'], num_processes=num_processes)
-    print('\nA total of %d citing institutions recorded.' % len(institutions))
+    author_institution_tuple_list = find_all_citing_institutions(author_profile['publications'], num_processes=num_processes)
+    print('\nA total of %d citing institutions recorded.\n' % len(author_institution_tuple_list))
 
-    coordinates = text_to_geocode(institutions)
+    if print_citing_institutions:
+        print('\nList of all citing authors and institutions before cleaning:\n')
+        for item in sorted(author_institution_tuple_list):
+            print(item)
+
+    cleaned_author_institution_tuple_list = clean_institution_names(author_institution_tuple_list)
+
+    if print_citing_institutions:
+        print('\nList of all citing authors and institutions after cleaning:\n')
+        for item in sorted(cleaned_author_institution_tuple_list):
+            print(item)
+
+    coordinates = institution_text_to_geocode(author_institution_tuple_list + cleaned_author_institution_tuple_list)
     print('\nConverted the institutions to Geocodes.')
 
     citation_map = create_map(coordinates, pin_colorful=pin_colorful)
     citation_map.save(output_path)
-    print('\nMap created and saved as citation_map.html.')
-
-    if print_citing_institutions:
-        print('\nList of all citing institutions:\n')
-        for institution in institutions:
-            print(institution)
+    print('\nMap created and saved at %s.' % output_path)
     return
 
 

{citation-map-2.0 → citation-map-3.1}/citation_map.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: citation-map
-Version: 2.0
+Version: 3.1
 Summary: A simple tool to generate your Google Scholar citation world map.
 Home-page: https://github.com/ChenLiu-1996/CitationMap
 Author: Chen Liu
@@ -39,8 +39,6 @@ If you open it on a browser, you will see your own version of the following cita
 
 <img src = "assets/citation_world_map.png" width=800>
 
-## News
-`citation-map` has been upgraded to version 2.0! I finally managed to drastically speed up the process using multiprocessing (without being blocked by Google Scholar). On my personal computer, processing my profile with 100 citations took 1 hour with version 1.0 while it's now taking 5 minutes with version 2.0.
 
 ## Usage
 1. Install the package.
@@ -88,6 +86,27 @@ If you open it on a browser, you will see your own version of the following cita
         If true, print the list of citing institutions (affiliations of citing authors).
     ```
 
+## Changelog
+
+### Version 3.0
+I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map. For example, the string "Yale University" is a much better entry than "Assistant Professor at Yale University".
+
+I applied a simple fix with some rule-based language processing. This helps us identify many missing citing locations.
+
+### Version 2.0
+I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
+
+On my personal computer, processing my profile with 100 citations took 1 hour with version 1.0 while it's now taking 5 minutes with version 2.0.
+
+With that said, please be careful and do not run this tool frequently. I can easily get on Google Scholar's blacklist after a few runs.
+
+### Version 1.0
+Very basic functionality.
+
+This script is a bit slow. On my personal computer, it takes half a minute to process each citation. If you have thousands of citations, it may or may not be a good idea to use this script.
+
+I tried to use multiprocessing, but unfortunately the excessive visits get me blocked by Google Scholar.
+
 ## Dependencies
 Dependencies (`scholarly`, `geopy`, `folium`, `tqdm`) are already taken care of when you install via pip.
 

{citation-map-2.0 → citation-map-3.1}/setup.py

@@ -5,7 +5,7 @@ with open('README.md') as f:
 
 setup(
     name='citation-map',
-    version='2.0',
+    version='3.1',
     license='MIT',
     author='Chen Liu',
     author_email='chen.liu.cl2482@yale.edu',