citation-map 4.2__tar.gz → 4.4__tar.gz

{citation-map-4.2 → citation-map-4.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: citation-map
-Version: 4.2
+Version: 4.4
 Summary: A simple tool to generate your Google Scholar citation world map.
 Home-page: https://github.com/ChenLiu-1996/CitationMap
 Author: Chen Liu
@@ -16,10 +16,11 @@ License-File: LICENSE.md
 
 # Google Scholar Citation World Map
 
+[![TechRxiv](https://img.shields.io/badge/TechRxiv-darkviolet)](https://www.techrxiv.org/users/809001/articles/1213717-citationmap-a-python-tool-to-identify-and-visualize-your-google-scholar-citations-around-the-world)
+[![OpenReview](https://img.shields.io/badge/OpenReview-firebrick)](https://openreview.net/pdf?id=BqJgCgl1IA)
 [![Latest PyPI version](https://img.shields.io/pypi/v/citation-map.svg)](https://pypi.org/project/citation-map/)
 [![PyPI download month](https://img.shields.io/pypi/dm/citation-map.svg)](https://pypistats.org/packages/citation-map)
 [![PyPI download day](https://img.shields.io/pypi/dd/citation-map.svg)](https://pypistats.org/packages/citation-map)
-[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
 [![CC BY-NC-SA 4.0][cc-by-nc-sa-shield]][cc-by-nc-sa]
 [![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa]
 
@@ -27,13 +28,14 @@ License-File: LICENSE.md
 [cc-by-nc-sa-image]: https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png
 [cc-by-nc-sa-shield]: https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg
 
-[Chen Liu](https://www.chenliu1996.com/), CS PhD Candidate at Yale University.
+[Chen Liu](https://chenliu-1996.github.io/), CS PhD Candidate at Yale University.
 
 Research areas: Machine Learning, Spatial-Temporal Modeling, Medical Vision, AI4Science.
 
+
 ## News
 **[Asking for advice]**
-1. This is my first time dealing with webscraping/crawling. Users have reported stability issues, and I suspect the major problems are (1) being caught by CAPTCHA or robot check, and (2) being flagged for blacklist by Google Scholar. If you are experienced in these areas and have good advice, I would highly appreciate a GitHub issue or a pull request.
+1. This is my first time dealing with webscraping/crawling. Users have reported stability issues, and I suspect the major problems are (1) being caught by CAPTCHA or robot check, and (2) being flagged for blacklist by Google Scholar. If you are experienced in these areas and have good advice, I would highly appreciate a **GitHub issue or a pull request**.
 
 [Aug 2, 2024] Version 4.0 released >>> Logic update. A new input argument `affiliation_conservative`. If set to True, we will use **a very conservative approach to identify affiliations** which leads to **much higher precision and lower recall**. Many thanks to [Zhijian Liu](https://github.com/zhijian-liu) for the [helpful discussion](https://github.com/ChenLiu-1996/CitationMap/issues/8).
 
@@ -63,6 +65,29 @@ Besides, there will be a **CSV file** recording citation information (citing aut
 
 **NOTE:** **Now you can trade off between affiliation precision and recall** with the `affiliation_conservative` option. If set to True, we will use the Google Scholar verified official organization name from the citing authors. This is a very conservative approach, since (1) the author needs to self-report it in the affiliation panel, (2) the author needs to verify with the matching email address, and (3) the organization needs to be recorded by Google Scholar. For example, Meta (the company) is not in the list. Many thanks to [Zhijian Liu](https://github.com/zhijian-liu) for the [helpful discussion](https://github.com/ChenLiu-1996/CitationMap/issues/8).
 
+## Citation
+BibTeX
+```
+@article{citationmap,
+  title={CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World},
+  author={Liu, Chen},
+  journal={Authorea Preprints},
+  year={2024},
+  publisher={Authorea}
+}
+```
+MLA
+```
+Liu, Chen. "CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World." Authorea Preprints (2024).
+```
+APA
+```
+Liu, C. (2024). CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World. Authorea Preprints.
+```
+Chicago
+```
+Liu, Chen. "CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World." Authorea Preprints (2024).
+```
 
 ## User Guide
 0. If you are new to Python, you probably want to start with a distribution of Python that also helps with environment management (such as [anaconda](https://www.anaconda.com/)). Once you have set up your environment (for example, when you reach the stage of `conda activate env39` in [this tutorial](https://www.youtube.com/watch?v=MUZtVEDKXsk&t=242s)), you can move on to the next step.
@@ -90,6 +115,8 @@ Besides, there will be a **CSV file** recording citation information (citing aut
         generate_citation_map(scholar_id)
     ```
 
+    Note that in Version 4.4, we support manual edit of the csv. You can run `generate_citation_map` first, and get the csv and HTML. Then, you can manually modify the csv content, and generate an updated HTML by running `generate_citation_map` again with `parse_csv=True`.
+
     Note that in Version 4.0, we will cache the results before identifying affiliations. So if you want to rerun the same author from scratch, you need to delete the cache (default location is 'cache').
 
     More input arguments are shown in [the demo script](https://github.com/ChenLiu-1996/CitationMap/blob/main/demo/demo.py).
@@ -140,10 +167,12 @@ Besides, there will be a **CSV file** recording citation information (citing aut
     - Some authors citing you do not have Google Scholar profiles.
     - Some authors citing you do not report their affiliations.
 2. Webscraping is performed, and CAPTCHA or robot check can often get us, especially if we crawl frequently. This is more often seen in highly cited users. Unless you are blocked by Google Scholar, at worst you will end up with missing several citing authors, which is not likely a huge deal for highly cited users anyways.
-3. `geopy.geocoders` is used to convert the citing authors' affiliations to geographic coordinates. To facilitate the process, I used some simple rule-based natural language processing to clean up the affiliations. As a result, you are expected to have:
-    - Underestimation if the affiliations are not found by `geopy.geocoders`.
-    - Underestimation if we experience communication error with `geopy.geocoders`.
-    - Overestimation if non-affiliation phrases are incorrectly identified as locations by `geopy.geocoders`. (Only relevant if `affiliation_conservative` is set to False.)
+3. Affiliation identification and geolocating issues. This is a joint effect between affiliation identification and geolocating. The number of citing affiliations will be:
+    - Underestimated if some affiliations are not found by `geopy.geocoders`.
+    - Underestimated if we experience communication errors with `geopy.geocoders`.
+    - (Aggressive approach only) Overestimated if non-affiliation phrases are incorrectly identified as locations by `geopy.geocoders`.
+    - (Conservative approach only) Underestimated if the citer did not verify with an email address under a matching affiliation domain.
+    - (Conservative approach only) Underestimated because all non-primary (with verified email address) affiliations are ignored.
 
     **Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
 
@@ -176,6 +205,16 @@ Besides, there will be a **CSV file** recording citation information (citing aut
       ```
 
 ## Changelog
+<details>
+<summary>Version 4.4 (Dec 6, 2024)</summary>
+<br>
+
+1. **Support manual edit of the csv.** Note you do need to manually correct the coordinates too if you choose to use this functionality. You can run `generate_citation_map` first, and get the csv and HTML. Then, you can manually modify the csv content, and generate an updated HTML by running `generate_citation_map` again with `parse_csv=True`.
+2. Slight optimization on caching.
+
+</details>
+
+
 <details>
 <summary>Version 4.0 (Aug 2, 2024)</summary>
 <br>

{citation-map-4.2 → citation-map-4.4}/README.md

@@ -1,9 +1,10 @@
 # Google Scholar Citation World Map
 
+[![TechRxiv](https://img.shields.io/badge/TechRxiv-darkviolet)](https://www.techrxiv.org/users/809001/articles/1213717-citationmap-a-python-tool-to-identify-and-visualize-your-google-scholar-citations-around-the-world)
+[![OpenReview](https://img.shields.io/badge/OpenReview-firebrick)](https://openreview.net/pdf?id=BqJgCgl1IA)
 [![Latest PyPI version](https://img.shields.io/pypi/v/citation-map.svg)](https://pypi.org/project/citation-map/)
 [![PyPI download month](https://img.shields.io/pypi/dm/citation-map.svg)](https://pypistats.org/packages/citation-map)
 [![PyPI download day](https://img.shields.io/pypi/dd/citation-map.svg)](https://pypistats.org/packages/citation-map)
-[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
 [![CC BY-NC-SA 4.0][cc-by-nc-sa-shield]][cc-by-nc-sa]
 [![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa]
 
@@ -11,13 +12,14 @@
 [cc-by-nc-sa-image]: https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png
 [cc-by-nc-sa-shield]: https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg
 
-[Chen Liu](https://www.chenliu1996.com/), CS PhD Candidate at Yale University.
+[Chen Liu](https://chenliu-1996.github.io/), CS PhD Candidate at Yale University.
 
 Research areas: Machine Learning, Spatial-Temporal Modeling, Medical Vision, AI4Science.
 
+
 ## News
 **[Asking for advice]**
-1. This is my first time dealing with webscraping/crawling. Users have reported stability issues, and I suspect the major problems are (1) being caught by CAPTCHA or robot check, and (2) being flagged for blacklist by Google Scholar. If you are experienced in these areas and have good advice, I would highly appreciate a GitHub issue or a pull request.
+1. This is my first time dealing with webscraping/crawling. Users have reported stability issues, and I suspect the major problems are (1) being caught by CAPTCHA or robot check, and (2) being flagged for blacklist by Google Scholar. If you are experienced in these areas and have good advice, I would highly appreciate a **GitHub issue or a pull request**.
 
 [Aug 2, 2024] Version 4.0 released >>> Logic update. A new input argument `affiliation_conservative`. If set to True, we will use **a very conservative approach to identify affiliations** which leads to **much higher precision and lower recall**. Many thanks to [Zhijian Liu](https://github.com/zhijian-liu) for the [helpful discussion](https://github.com/ChenLiu-1996/CitationMap/issues/8).
 
@@ -47,6 +49,29 @@ Besides, there will be a **CSV file** recording citation information (citing aut
 
 **NOTE:** **Now you can trade off between affiliation precision and recall** with the `affiliation_conservative` option. If set to True, we will use the Google Scholar verified official organization name from the citing authors. This is a very conservative approach, since (1) the author needs to self-report it in the affiliation panel, (2) the author needs to verify with the matching email address, and (3) the organization needs to be recorded by Google Scholar. For example, Meta (the company) is not in the list. Many thanks to [Zhijian Liu](https://github.com/zhijian-liu) for the [helpful discussion](https://github.com/ChenLiu-1996/CitationMap/issues/8).
 
+## Citation
+BibTeX
+```
+@article{citationmap,
+  title={CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World},
+  author={Liu, Chen},
+  journal={Authorea Preprints},
+  year={2024},
+  publisher={Authorea}
+}
+```
+MLA
+```
+Liu, Chen. "CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World." Authorea Preprints (2024).
+```
+APA
+```
+Liu, C. (2024). CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World. Authorea Preprints.
+```
+Chicago
+```
+Liu, Chen. "CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World." Authorea Preprints (2024).
+```
 
 ## User Guide
 0. If you are new to Python, you probably want to start with a distribution of Python that also helps with environment management (such as [anaconda](https://www.anaconda.com/)). Once you have set up your environment (for example, when you reach the stage of `conda activate env39` in [this tutorial](https://www.youtube.com/watch?v=MUZtVEDKXsk&t=242s)), you can move on to the next step.
@@ -74,6 +99,8 @@ Besides, there will be a **CSV file** recording citation information (citing aut
         generate_citation_map(scholar_id)
     ```
 
+    Note that in Version 4.4, we support manual edit of the csv. You can run `generate_citation_map` first, and get the csv and HTML. Then, you can manually modify the csv content, and generate an updated HTML by running `generate_citation_map` again with `parse_csv=True`.
+
     Note that in Version 4.0, we will cache the results before identifying affiliations. So if you want to rerun the same author from scratch, you need to delete the cache (default location is 'cache').
 
     More input arguments are shown in [the demo script](https://github.com/ChenLiu-1996/CitationMap/blob/main/demo/demo.py).
@@ -124,10 +151,12 @@ Besides, there will be a **CSV file** recording citation information (citing aut
     - Some authors citing you do not have Google Scholar profiles.
     - Some authors citing you do not report their affiliations.
 2. Webscraping is performed, and CAPTCHA or robot check can often get us, especially if we crawl frequently. This is more often seen in highly cited users. Unless you are blocked by Google Scholar, at worst you will end up with missing several citing authors, which is not likely a huge deal for highly cited users anyways.
-3. `geopy.geocoders` is used to convert the citing authors' affiliations to geographic coordinates. To facilitate the process, I used some simple rule-based natural language processing to clean up the affiliations. As a result, you are expected to have:
-    - Underestimation if the affiliations are not found by `geopy.geocoders`.
-    - Underestimation if we experience communication error with `geopy.geocoders`.
-    - Overestimation if non-affiliation phrases are incorrectly identified as locations by `geopy.geocoders`. (Only relevant if `affiliation_conservative` is set to False.)
+3. Affiliation identification and geolocating issues. This is a joint effect between affiliation identification and geolocating. The number of citing affiliations will be:
+    - Underestimated if some affiliations are not found by `geopy.geocoders`.
+    - Underestimated if we experience communication errors with `geopy.geocoders`.
+    - (Aggressive approach only) Overestimated if non-affiliation phrases are incorrectly identified as locations by `geopy.geocoders`.
+    - (Conservative approach only) Underestimated if the citer did not verify with an email address under a matching affiliation domain.
+    - (Conservative approach only) Underestimated because all non-primary (with verified email address) affiliations are ignored.
 
     **Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
 
@@ -160,6 +189,16 @@ Besides, there will be a **CSV file** recording citation information (citing aut
       ```
 
 ## Changelog
+<details>
+<summary>Version 4.4 (Dec 6, 2024)</summary>
+<br>
+
+1. **Support manual edit of the csv.** Note you do need to manually correct the coordinates too if you choose to use this functionality. You can run `generate_citation_map` first, and get the csv and HTML. Then, you can manually modify the csv content, and generate an updated HTML by running `generate_citation_map` again with `parse_csv=True`.
+2. Slight optimization on caching.
+
+</details>
+
+
 <details>
 <summary>Version 4.0 (Aug 2, 2024)</summary>
 <br>

{citation-map-4.2 → citation-map-4.4}/citation_map/citation_map.py

@@ -14,14 +14,15 @@ from geopy.geocoders import Nominatim
 from multiprocessing import Pool
 from scholarly import scholarly, ProxyGenerator
 from tqdm import tqdm
-from typing import Any, Iterable, List, Tuple
+from typing import Any, List, Tuple
 
 from .scholarly_support import get_citing_author_ids_and_citing_papers, get_organization_name
 
 
 def find_all_citing_authors(scholar_id: str, num_processes: int = 16) -> List[Tuple[str]]:
     '''
-    Step 1. Find all citing authors.
+    Step 1. Find all publications of the given Google Scholar ID.
+    Step 2. Find all citing authors.
     '''
     # Find Google Scholar Profile using Scholar ID.
     author = scholarly.search_author_id(scholar_id)
@@ -70,7 +71,7 @@ def find_all_citing_affiliations(all_citing_author_paper_tuple_list: List[Tuple[
                                  num_processes: int = 16,
                                  affiliation_conservative: bool = False):
     '''
-    Step 2. Find all citing affiliations.
+    Step 3. Find all citing affiliations.
     '''
     if affiliation_conservative:
         __affiliations_from_authors = __affiliations_from_authors_conservative
@@ -96,7 +97,7 @@ def find_all_citing_affiliations(all_citing_author_paper_tuple_list: List[Tuple[
 
 def clean_affiliation_names(author_paper_affiliation_tuple_list: List[Tuple[str]]) -> List[Tuple[str]]:
     '''
-    Step 3 (Optional). Clean up the names of affiliations from the authors' affiliation tab on their Google Scholar profiles.
+    Optional Step. Clean up the names of affiliations from the authors' affiliation tab on their Google Scholar profiles.
     NOTE: This logic is very naive. Please send an issue or pull request if you have any idea how to improve it.
     Currently we will not consider any paid service or tools that pose extra burden on the users, such as GPT API.
     '''
@@ -177,9 +178,32 @@ def affiliation_text_to_geocode(author_paper_affiliation_tuple_list: List[Tuple[
     coordinates_and_info = [item for item in coordinates_and_info if item is not None]  # Filter out empty entries.
     return coordinates_and_info
 
+def export_dict_to_csv(coordinates_and_info: List[Tuple[str]], csv_output_path: str) -> None:
+    '''
+    Step 5.1: Export csv file recording citation information.
+    '''
+
+    citation_df = pd.DataFrame(coordinates_and_info,
+                               columns=['citing author name', 'citing paper title', 'cited paper title',
+                                        'affiliation', 'latitude', 'longitude',
+                                        'county', 'city', 'state', 'country'])
+
+    citation_df.to_csv(csv_output_path)
+    return
+
+def read_csv_to_dict(csv_path: str) -> None:
+    '''
+    Step 5.1: Read csv file recording citation information.
+    Only relevant if `read_from_csv` is True.
+    '''
+
+    citation_df = pd.read_csv(csv_path, index_col=0)
+    coordinates_and_info = list(citation_df.itertuples(index=False, name=None))
+    return coordinates_and_info
+
 def create_map(coordinates_and_info: List[Tuple[str]], pin_colorful: bool = True):
     '''
-    Step 5.1: Create the Citation World Map.
+    Step 5.2: Create the Citation World Map.
 
     For authors under the same affiliations, they will be displayed in the same pin.
     '''
@@ -217,18 +241,6 @@ def create_map(coordinates_and_info: List[Tuple[str]], pin_colorful: bool = True
             folium.Marker([lat, lon], popup='%s (%s)' % (affiliation_name, ' & '.join(author_name_list))).add_to(citation_map)
     return citation_map
 
-def export_csv(coordinates_and_info: List[Tuple[str]], csv_output_path: str) -> None:
-    '''
-    Step 5.2: Export csv file recording citation information.
-    '''
-
-    citation_df = pd.DataFrame(coordinates_and_info,
-                               columns=['citing author name', 'citing paper title', 'cited paper title',
-                                        'affiliation', 'latitude', 'longitude',
-                                        'county', 'city', 'state', 'country'])
-
-    citation_df.to_csv(csv_output_path)
-    return
 
 def __fill_publication_metadata(pub):
     time.sleep(random.uniform(1, 5))  # Random delay to reduce risk of being blocked.
@@ -328,6 +340,7 @@ def load_cache(fpath: str) -> Any:
 def generate_citation_map(scholar_id: str,
                           output_path: str = 'citation_map.html',
                           csv_output_path: str = 'citation_info.csv',
+                          parse_csv: bool = False,
                           cache_folder: str = 'cache',
                           affiliation_conservative: bool = False,
                           num_processes: int = 16,
@@ -347,6 +360,9 @@ def generate_citation_map(scholar_id: str,
     csv_output_path: str
         (default is 'citation_info.csv')
         The path to the output csv file.
+    parse_csv: bool
+        (default is False)
+        If True, will directly jump to Step 5.2, using the information loaded from the csv.
     cache_folder: str
         (default is 'cache')
         The folder to save intermediate results, after finding (author, paper) but before finding the affiliations.
@@ -372,83 +388,110 @@ def generate_citation_map(scholar_id: str,
         If true, print the list of citing affiliations (affiliations of citing authors).
     '''
 
-    if use_proxy:
-        pg = ProxyGenerator()
-        pg.FreeProxies()
-        scholarly.use_proxy(pg)
-        print('Using proxy.')
+    if not parse_csv:
 
-    if cache_folder is not None:
-        cache_path = os.path.join(cache_folder, scholar_id, 'all_citing_author_paper_tuple_list.pkl')
-    else:
-        cache_path = None
+        if use_proxy:
+            pg = ProxyGenerator()
+            pg.FreeProxies()
+            scholarly.use_proxy(pg)
+            print('Using proxy.')
 
-    if cache_path is None or not os.path.exists(cache_path):
-        print('No cache found for this author. Running from scratch.\n')
+        if cache_folder is not None:
+            cache_path = os.path.join(cache_folder, scholar_id, 'all_citing_author_paper_tuple_list.pkl')
+        else:
+            cache_path = None
 
-        # NOTE: Step 1. Find all citing authors.
-        all_citing_author_paper_tuple_list = find_all_citing_authors(scholar_id=scholar_id,
-                                                                     num_processes=num_processes)
-        print('A total of %d citing authors recorded.\n' % len(all_citing_author_paper_tuple_list))
-        if cache_path is not None and len(all_citing_author_paper_tuple_list) > 0:
-            save_cache(all_citing_author_paper_tuple_list, cache_path)
-        print('Saved to cache: %s.\n' % cache_path)
+        if cache_path is None or not os.path.exists(cache_path):
+            print('No cache found for this author. Finding citing authors from scratch.\n')
 
-    else:
-        print('Cache found. Loading author paper affiliation information from cache.\n')
-
-        # NOTE: Step 1. Load all citing authors.
-        all_citing_author_paper_tuple_list = load_cache(cache_path)
-        print('Loaded from cache: %s.\n' % cache_path)
-        print('A total of %d citing authors loaded.\n' % len(all_citing_author_paper_tuple_list))
-
-    # NOTE: Step 2. Find all citing affiliations.
-    print('Identifying affiliations using the %s approach.' % ('conservative' if affiliation_conservative else 'aggressive'))
-    author_paper_affiliation_tuple_list = find_all_citing_affiliations(all_citing_author_paper_tuple_list,
-                                                                       num_processes=num_processes,
-                                                                       affiliation_conservative=affiliation_conservative)
-    print('\nA total of %d citing affiliations recorded.\n' % len(author_paper_affiliation_tuple_list))
-    # Take unique tuples.
-    author_paper_affiliation_tuple_list = list(set(author_paper_affiliation_tuple_list))
+            # NOTE: Step 1. Find all publications of the given Google Scholar ID.
+            #       Step 2. Find all citing authors.
+            all_citing_author_paper_tuple_list = find_all_citing_authors(scholar_id=scholar_id,
+                                                                        num_processes=num_processes)
+            print('A total of %d citing authors recorded.\n' % len(all_citing_author_paper_tuple_list))
+            if cache_path is not None and len(all_citing_author_paper_tuple_list) > 0:
+                save_cache(all_citing_author_paper_tuple_list, cache_path)
+            print('Saved to cache: %s.\n' % cache_path)
+
+        else:
+            print('Cache found. Loading author paper information from cache.\n')
+            all_citing_author_paper_tuple_list = load_cache(cache_path)
+            print('Loaded from cache: %s.\n' % cache_path)
+            print('A total of %d citing authors loaded.\n' % len(all_citing_author_paper_tuple_list))
+
+        if cache_folder is not None:
+            cache_path = os.path.join(cache_folder, scholar_id, 'author_paper_affiliation_tuple_list.pkl')
+        else:
+            cache_path = None
+
+        if cache_path is None or not os.path.exists(cache_path):
+            print('No cache found for this author. Finding citing affiliations from scratch.\n')
+
+            # NOTE: Step 2. Find all citing affiliations.
+            print('Identifying affiliations using the %s approach.' % ('conservative' if affiliation_conservative else 'aggressive'))
+            author_paper_affiliation_tuple_list = find_all_citing_affiliations(all_citing_author_paper_tuple_list,
+                                                                            num_processes=num_processes,
+                                                                            affiliation_conservative=affiliation_conservative)
+            print('\nA total of %d citing affiliations recorded.\n' % len(author_paper_affiliation_tuple_list))
+            # Take unique tuples.
+            author_paper_affiliation_tuple_list = list(set(author_paper_affiliation_tuple_list))
+
+            # NOTE: Step 3. Clean the affiliation strings (optional, only used if taking the aggressive approach).
+            if print_citing_affiliations:
+                if affiliation_conservative:
+                    print('Taking the conservative approach. Will not need to clean the affiliation names.')
+                    print('List of all citing authors and affiliations:\n')
+                else:
+                    print('Taking the aggressive approach. Cleaning the affiliation names.')
+                    print('List of all citing authors and affiliations before cleaning:\n')
+                __print_author_and_affiliation(author_paper_affiliation_tuple_list)
+            if not affiliation_conservative:
+                cleaned_author_paper_affiliation_tuple_list = clean_affiliation_names(author_paper_affiliation_tuple_list)
+                if print_citing_affiliations:
+                    print('List of all citing authors and affiliations after cleaning:\n')
+                    __print_author_and_affiliation(cleaned_author_paper_affiliation_tuple_list)
+                # Use the merged set to maximize coverage.
+                author_paper_affiliation_tuple_list += cleaned_author_paper_affiliation_tuple_list
+                # Take unique tuples.
+                author_paper_affiliation_tuple_list = list(set(author_paper_affiliation_tuple_list))
+
+            if cache_path is not None and len(author_paper_affiliation_tuple_list) > 0:
+                save_cache(author_paper_affiliation_tuple_list, cache_path)
+            print('Saved to cache: %s.\n' % cache_path)
 
-    # NOTE: Step 3. Clean the affiliation strings (optional, only used if taking the aggressive approach).
-    if print_citing_affiliations:
-        if affiliation_conservative:
-            print('Taking the conservative approach. Will not need to clean the affiliation names.')
-            print('List of all citing authors and affiliations:\n')
         else:
-            print('Taking the aggressive approach. Cleaning the affiliation names.')
-            print('List of all citing authors and affiliations before cleaning:\n')
-        __print_author_and_affiliation(author_paper_affiliation_tuple_list)
-    if not affiliation_conservative:
-        cleaned_author_paper_affiliation_tuple_list = clean_affiliation_names(author_paper_affiliation_tuple_list)
-        if print_citing_affiliations:
-            print('List of all citing authors and affiliations after cleaning:\n')
-            __print_author_and_affiliation(cleaned_author_paper_affiliation_tuple_list)
-        # Use the merged set to maximize coverage.
-        author_paper_affiliation_tuple_list += cleaned_author_paper_affiliation_tuple_list
+            print('Cache found. Loading author paper and affiliation information from cache.\n')
+            author_paper_affiliation_tuple_list = load_cache(cache_path)
+            print('List of all citing authors and affiliations loaded:\n')
+            __print_author_and_affiliation(author_paper_affiliation_tuple_list)
+
+        # NOTE: Step 4. Convert affiliations in plain text to Geocode.
+        coordinates_and_info = affiliation_text_to_geocode(author_paper_affiliation_tuple_list)
         # Take unique tuples.
-        author_paper_affiliation_tuple_list = list(set(author_paper_affiliation_tuple_list))
+        coordinates_and_info = list(set(coordinates_and_info))
 
-    # NOTE: Step 4. Convert affiliations in plain text to Geocode.
-    coordinates_and_info = affiliation_text_to_geocode(author_paper_affiliation_tuple_list)
-    # Take unique tuples.
-    coordinates_and_info = list(set(coordinates_and_info))
+        # NOTE: Step 5.1. Export csv file recording citation information.
+        export_dict_to_csv(coordinates_and_info, csv_output_path)
+        print('\nCitation information exported to %s.' % csv_output_path)
 
-    # NOTE: Step 5.1. Create the citation world map.
+    else:
+        print('\nDirectly parsing the csv. Skipping all previous steps.')
+        assert os.path.isfile(csv_output_path), '`csv_output_path` is not a file.'
+        coordinates_and_info = read_csv_to_dict(csv_output_path)
+        print('\nCitation information loaded from %s.' % csv_output_path)
+
+    # NOTE: Step 5.2. Create the citation world map.
     citation_map = create_map(coordinates_and_info, pin_colorful=pin_colorful)
     citation_map.save(output_path)
-    print('\nMap created and saved at %s.\n' % output_path)
-
-    # NOTE: Step 5.2. Export csv file recording citation information.
-    export_csv(coordinates_and_info, csv_output_path)
-    print('\nCitation information exported to %s.' % csv_output_path)
+    print('\nHTML map created and saved at %s.\n' % output_path)
     return
 
 
 if __name__ == '__main__':
     # Replace this with your Google Scholar ID.
     scholar_id = '3rDjnykAAAAJ'
-    generate_citation_map(scholar_id, output_path='citation_map.html', csv_output_path='citation_info.csv',
+    generate_citation_map(scholar_id, output_path='citation_map.html',
+                          csv_output_path='citation_info.csv',
+                          parse_csv=False,
                           cache_folder='cache', affiliation_conservative=True, num_processes=16,
                           use_proxy=False, pin_colorful=True, print_citing_affiliations=True)

{citation-map-4.2 → citation-map-4.4}/citation_map/scholarly_support.py

@@ -7,6 +7,35 @@ from bs4 import BeautifulSoup
 from typing import List
 
 
+def get_html_per_citation_page(soup) -> List[str]:
+    '''
+    Utility to query each page containing results for
+    cited work.
+    Parameters
+    --------
+    soup: Beautiful Soup object pointing to current page.
+    '''
+    citing_authors_and_citing_papers = []
+
+    for result in soup.find_all('div', class_='gs_ri'):
+        title_tag = result.find('h3', class_='gs_rt')
+        if title_tag:
+            paper_parsed = False
+            author_links = result.find_all('a', href=True)
+            title_text = title_tag.get_text()
+            title = title_text.replace('[HTML]', '').replace('[PDF]', '')
+            for link in author_links:
+                if 'user=' in link['href']:
+                    author_id = link['href'].split('user=')[1].split('&')[0]
+                    citing_authors_and_citing_papers.append((author_id, title))
+                    paper_parsed = True
+            if not paper_parsed:
+                print("[WARNING!] Could not find author links for ", title)
+        else:
+            continue
+    return citing_authors_and_citing_papers
+
+
 def get_citing_author_ids_and_citing_papers(paper_url: str) -> List[str]:
     '''
     Find the (Google Scholar IDs of authors, titles of papers) who cite a given paper on Google Scholar.
@@ -15,7 +44,6 @@ def get_citing_author_ids_and_citing_papers(paper_url: str) -> List[str]:
     --------
     paper_url: URL of the paper BEING cited.
     '''
-
     citing_authors_and_citing_papers = []
 
     headers = requests.utils.default_headers()
@@ -44,36 +72,25 @@ def get_citing_author_ids_and_citing_papers(paper_url: str) -> List[str]:
 
     # Loop through the citation results and find citing authors and papers.
     current_page_number = 1
-    while True:
-        for result in soup.find_all('div', class_='gs_ri'):
-            title_tag = result.find('h3', class_='gs_rt')
-            if title_tag:
-                author_links = result.find_all('a', href=True)
-                title = title_tag.get_text().replace('[HTML]', '').replace('[PDF]', '')
-                for link in author_links:
-                    if 'user=' in link['href']:
-                        author_id = link['href'].split('user=')[1].split('&')[0]
-                        citing_authors_and_citing_papers.append((author_id, title))
-            else:
-                continue
-
-        # Find the page navigation.
-        navigation_buttons = soup.find_all('a', class_='gs_nma')
-        for navigation in navigation_buttons:
-            page_number_str = navigation.text
-            if page_number_str and page_number_str.isnumeric() and int(page_number_str) == current_page_number + 1:
-                # Found the correct button for next page.
-                current_page_number += 1
-                next_url = 'https://scholar.google.com' + navigation['href']
-                time.sleep(random.uniform(1, 5))  # Random delay to reduce risk of being blocked.
-                response = requests.get(next_url, headers=headers)
-                if response.status_code != 200:
-                    break
-                soup = BeautifulSoup(response.text, 'html.parser')
-            else:
-                continue
+    citing_authors_and_citing_papers += get_html_per_citation_page(soup)
+
+    # Find the page navigation.
+    navigation_buttons = soup.find_all('a', class_='gs_nma')
+    for navigation in navigation_buttons:
+        page_number_str = navigation.text
+        if page_number_str and page_number_str.isnumeric() and int(page_number_str) == current_page_number + 1:
+            # Found the correct button for next page.
+            current_page_number += 1
+            next_url = 'https://scholar.google.com' + navigation['href']
+            time.sleep(random.uniform(1, 5))  # Random delay to reduce risk of being blocked.
+
+            response = requests.get(next_url, headers=headers)
+            if response.status_code != 200:
+                break
+            soup = BeautifulSoup(response.text, 'html.parser')
+            citing_authors_and_citing_papers += get_html_per_citation_page(soup)
         else:
-            break
+            continue
 
     return citing_authors_and_citing_papers
 

{citation-map-4.2 → citation-map-4.4}/citation_map.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: citation-map
-Version: 4.2
+Version: 4.4
 Summary: A simple tool to generate your Google Scholar citation world map.
 Home-page: https://github.com/ChenLiu-1996/CitationMap
 Author: Chen Liu
@@ -16,10 +16,11 @@ License-File: LICENSE.md
 
 # Google Scholar Citation World Map
 
+[![TechRxiv](https://img.shields.io/badge/TechRxiv-darkviolet)](https://www.techrxiv.org/users/809001/articles/1213717-citationmap-a-python-tool-to-identify-and-visualize-your-google-scholar-citations-around-the-world)
+[![OpenReview](https://img.shields.io/badge/OpenReview-firebrick)](https://openreview.net/pdf?id=BqJgCgl1IA)
 [![Latest PyPI version](https://img.shields.io/pypi/v/citation-map.svg)](https://pypi.org/project/citation-map/)
 [![PyPI download month](https://img.shields.io/pypi/dm/citation-map.svg)](https://pypistats.org/packages/citation-map)
 [![PyPI download day](https://img.shields.io/pypi/dd/citation-map.svg)](https://pypistats.org/packages/citation-map)
-[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
 [![CC BY-NC-SA 4.0][cc-by-nc-sa-shield]][cc-by-nc-sa]
 [![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa]
 
@@ -27,13 +28,14 @@ License-File: LICENSE.md
 [cc-by-nc-sa-image]: https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png
 [cc-by-nc-sa-shield]: https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg
 
-[Chen Liu](https://www.chenliu1996.com/), CS PhD Candidate at Yale University.
+[Chen Liu](https://chenliu-1996.github.io/), CS PhD Candidate at Yale University.
 
 Research areas: Machine Learning, Spatial-Temporal Modeling, Medical Vision, AI4Science.
 
+
 ## News
 **[Asking for advice]**
-1. This is my first time dealing with webscraping/crawling. Users have reported stability issues, and I suspect the major problems are (1) being caught by CAPTCHA or robot check, and (2) being flagged for blacklist by Google Scholar. If you are experienced in these areas and have good advice, I would highly appreciate a GitHub issue or a pull request.
+1. This is my first time dealing with webscraping/crawling. Users have reported stability issues, and I suspect the major problems are (1) being caught by CAPTCHA or robot check, and (2) being flagged for blacklist by Google Scholar. If you are experienced in these areas and have good advice, I would highly appreciate a **GitHub issue or a pull request**.
 
 [Aug 2, 2024] Version 4.0 released >>> Logic update. A new input argument `affiliation_conservative`. If set to True, we will use **a very conservative approach to identify affiliations** which leads to **much higher precision and lower recall**. Many thanks to [Zhijian Liu](https://github.com/zhijian-liu) for the [helpful discussion](https://github.com/ChenLiu-1996/CitationMap/issues/8).
 
@@ -63,6 +65,29 @@ Besides, there will be a **CSV file** recording citation information (citing aut
 
 **NOTE:** **Now you can trade off between affiliation precision and recall** with the `affiliation_conservative` option. If set to True, we will use the Google Scholar verified official organization name from the citing authors. This is a very conservative approach, since (1) the author needs to self-report it in the affiliation panel, (2) the author needs to verify with the matching email address, and (3) the organization needs to be recorded by Google Scholar. For example, Meta (the company) is not in the list. Many thanks to [Zhijian Liu](https://github.com/zhijian-liu) for the [helpful discussion](https://github.com/ChenLiu-1996/CitationMap/issues/8).
 
+## Citation
+BibTeX
+```
+@article{citationmap,
+  title={CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World},
+  author={Liu, Chen},
+  journal={Authorea Preprints},
+  year={2024},
+  publisher={Authorea}
+}
+```
+MLA
+```
+Liu, Chen. "CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World." Authorea Preprints (2024).
+```
+APA
+```
+Liu, C. (2024). CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World. Authorea Preprints.
+```
+Chicago
+```
+Liu, Chen. "CitationMap: A Python Tool to Identify and Visualize Your Google Scholar Citations Around the World." Authorea Preprints (2024).
+```
 
 ## User Guide
 0. If you are new to Python, you probably want to start with a distribution of Python that also helps with environment management (such as [anaconda](https://www.anaconda.com/)). Once you have set up your environment (for example, when you reach the stage of `conda activate env39` in [this tutorial](https://www.youtube.com/watch?v=MUZtVEDKXsk&t=242s)), you can move on to the next step.
@@ -90,6 +115,8 @@ Besides, there will be a **CSV file** recording citation information (citing aut
         generate_citation_map(scholar_id)
     ```
 
+    Note that in Version 4.4, we support manual edit of the csv. You can run `generate_citation_map` first, and get the csv and HTML. Then, you can manually modify the csv content, and generate an updated HTML by running `generate_citation_map` again with `parse_csv=True`.
+
     Note that in Version 4.0, we will cache the results before identifying affiliations. So if you want to rerun the same author from scratch, you need to delete the cache (default location is 'cache').
 
     More input arguments are shown in [the demo script](https://github.com/ChenLiu-1996/CitationMap/blob/main/demo/demo.py).
@@ -140,10 +167,12 @@ Besides, there will be a **CSV file** recording citation information (citing aut
     - Some authors citing you do not have Google Scholar profiles.
     - Some authors citing you do not report their affiliations.
 2. Webscraping is performed, and CAPTCHA or robot check can often get us, especially if we crawl frequently. This is more often seen in highly cited users. Unless you are blocked by Google Scholar, at worst you will end up with missing several citing authors, which is not likely a huge deal for highly cited users anyways.
-3. `geopy.geocoders` is used to convert the citing authors' affiliations to geographic coordinates. To facilitate the process, I used some simple rule-based natural language processing to clean up the affiliations. As a result, you are expected to have:
-    - Underestimation if the affiliations are not found by `geopy.geocoders`.
-    - Underestimation if we experience communication error with `geopy.geocoders`.
-    - Overestimation if non-affiliation phrases are incorrectly identified as locations by `geopy.geocoders`. (Only relevant if `affiliation_conservative` is set to False.)
+3. Affiliation identification and geolocating issues. This is a joint effect between affiliation identification and geolocating. The number of citing affiliations will be:
+    - Underestimated if some affiliations are not found by `geopy.geocoders`.
+    - Underestimated if we experience communication errors with `geopy.geocoders`.
+    - (Aggressive approach only) Overestimated if non-affiliation phrases are incorrectly identified as locations by `geopy.geocoders`.
+    - (Conservative approach only) Underestimated if the citer did not verify with an email address under a matching affiliation domain.
+    - (Conservative approach only) Underestimated because all non-primary (with verified email address) affiliations are ignored.
 
     **Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
 
@@ -176,6 +205,16 @@ Besides, there will be a **CSV file** recording citation information (citing aut
       ```
 
 ## Changelog
+<details>
+<summary>Version 4.4 (Dec 6, 2024)</summary>
+<br>
+
+1. **Support manual edit of the csv.** Note you do need to manually correct the coordinates too if you choose to use this functionality. You can run `generate_citation_map` first, and get the csv and HTML. Then, you can manually modify the csv content, and generate an updated HTML by running `generate_citation_map` again with `parse_csv=True`.
+2. Slight optimization on caching.
+
+</details>
+
+
 <details>
 <summary>Version 4.0 (Aug 2, 2024)</summary>
 <br>

{citation-map-4.2 → citation-map-4.4}/setup.py

@@ -5,7 +5,7 @@ with open('README.md') as f:
 
 setup(
     name='citation-map',
-    version='4.2',
+    version='4.4',
     license='CC BY-NC-SA',
     author='Chen Liu',
     author_email='chen.liu.cl2482@yale.edu',