citation-map 3.3__tar.gz → 3.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: citation-map
3
- Version: 3.3
3
+ Version: 3.5
4
4
  Summary: A simple tool to generate your Google Scholar citation world map.
5
5
  Home-page: https://github.com/ChenLiu-1996/CitationMap
6
6
  Author: Chen Liu
@@ -17,7 +17,6 @@ License-File: LICENSE.md
17
17
  # Google Scholar Citation World Map
18
18
 
19
19
  [![Latest PyPI version](https://img.shields.io/pypi/v/citation-map.svg)](https://pypi.org/project/citation-map/)
20
- [![PyPI license](https://img.shields.io/pypi/l/citation-map.svg)](https://pypi.python.org/pypi/citation-map/)
21
20
  [![PyPI download month](https://img.shields.io/pypi/dm/citation-map.svg)](https://pypi.python.org/pypi/citation-map/)
22
21
  [![PyPI download day](https://img.shields.io/pypi/dd/citation-map.svg)](https://pypi.python.org/pypi/citation-map/)
23
22
  [![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
@@ -32,6 +31,8 @@ This is a simple Python tool to generate an HTML citation world map from your Go
32
31
 
33
32
  It is easy to install (`pip install citation-map`, available on [PyPI](https://pypi.org/project/citation-map/)) and easy to use (see the [Usage](https://github.com/ChenLiu-1996/CitationMap?tab=readme-ov-file#usage) section).
34
33
 
34
+ **You don't need to fork this repo unless you want to make custom changes.**
35
+
35
36
  ## Expected Outcome
36
37
  You will be given an HTML file as the output of the script.
37
38
 
@@ -45,6 +46,10 @@ If you open it on a browser, you will see your own version of the following cita
45
46
  ```
46
47
  pip install citation-map
47
48
  ```
49
+ Remember to upgrade to the latest version if you installed an earlier version.
50
+ ```
51
+ pip install citation-map --upgrade
52
+ ```
48
53
 
49
54
  2. Find your Google Scholar ID.
50
55
 
@@ -70,9 +75,14 @@ If you open it on a browser, you will see your own version of the following cita
70
75
  output_path: str
71
76
  (default is 'citation_map.html')
72
77
  The path to the output HTML file.
73
- num_processes: int
78
+ num_processes_major: int
74
79
  (default is 16)
75
- Number of separate processes for parallel processing.
80
+ Number of processes for parallel processing.
81
+ Related to `scholarly.citedby()`. This causes Google Scholar blocks if set too high.
82
+ num_processes_minor: int
83
+ (default is 32)
84
+ Number of processes for parallel processing.
85
+ Related to all other processes. Usually not causing problems.
76
86
  use_proxy: bool
77
87
  (default is False)
78
88
  If true, we will use a scholarly proxy.
@@ -86,6 +96,13 @@ If you open it on a browser, you will see your own version of the following cita
86
96
  If true, print the list of citing institutions (affiliations of citing authors).
87
97
  ```
88
98
 
99
+ ## Debug
100
+ 1. `scholarly._proxy_generator.MaxTriesExceededException`
101
+
102
+ - From my experience, this is a good indicator that your IP address is blocked by Google Scholar due to excessive or frequent crawling (using the `scholarly` package).
103
+ - One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
104
+ - In case this does not help, you can try to reduce the number of processes (e.g., setting `num_processes_major=1`).
105
+
89
106
  ## Changelog
90
107
 
91
108
  ### Version 3.0
@@ -1,7 +1,6 @@
1
1
  # Google Scholar Citation World Map
2
2
 
3
3
  [![Latest PyPI version](https://img.shields.io/pypi/v/citation-map.svg)](https://pypi.org/project/citation-map/)
4
- [![PyPI license](https://img.shields.io/pypi/l/citation-map.svg)](https://pypi.python.org/pypi/citation-map/)
5
4
  [![PyPI download month](https://img.shields.io/pypi/dm/citation-map.svg)](https://pypi.python.org/pypi/citation-map/)
6
5
  [![PyPI download day](https://img.shields.io/pypi/dd/citation-map.svg)](https://pypi.python.org/pypi/citation-map/)
7
6
  [![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
@@ -16,6 +15,8 @@ This is a simple Python tool to generate an HTML citation world map from your Go
16
15
 
17
16
  It is easy to install (`pip install citation-map`, available on [PyPI](https://pypi.org/project/citation-map/)) and easy to use (see the [Usage](https://github.com/ChenLiu-1996/CitationMap?tab=readme-ov-file#usage) section).
18
17
 
18
+ **You don't need to fork this repo unless you want to make custom changes.**
19
+
19
20
  ## Expected Outcome
20
21
  You will be given an HTML file as the output of the script.
21
22
 
@@ -29,6 +30,10 @@ If you open it on a browser, you will see your own version of the following cita
29
30
  ```
30
31
  pip install citation-map
31
32
  ```
33
+ Remember to upgrade to the latest version if you installed an earlier version.
34
+ ```
35
+ pip install citation-map --upgrade
36
+ ```
32
37
 
33
38
  2. Find your Google Scholar ID.
34
39
 
@@ -54,9 +59,14 @@ If you open it on a browser, you will see your own version of the following cita
54
59
  output_path: str
55
60
  (default is 'citation_map.html')
56
61
  The path to the output HTML file.
57
- num_processes: int
62
+ num_processes_major: int
58
63
  (default is 16)
59
- Number of separate processes for parallel processing.
64
+ Number of processes for parallel processing.
65
+ Related to `scholarly.citedby()`. This causes Google Scholar blocks if set too high.
66
+ num_processes_minor: int
67
+ (default is 32)
68
+ Number of processes for parallel processing.
69
+ Related to all other processes. Usually not causing problems.
60
70
  use_proxy: bool
61
71
  (default is False)
62
72
  If true, we will use a scholarly proxy.
@@ -70,6 +80,13 @@ If you open it on a browser, you will see your own version of the following cita
70
80
  If true, print the list of citing institutions (affiliations of citing authors).
71
81
  ```
72
82
 
83
+ ## Debug
84
+ 1. `scholarly._proxy_generator.MaxTriesExceededException`
85
+
86
+ - From my experience, this is a good indicator that your IP address is blocked by Google Scholar due to excessive or frequent crawling (using the `scholarly` package).
87
+ - One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
88
+ - In case this does not help, you can try to reduce the number of processes (e.g., setting `num_processes_major=1`).
89
+
73
90
  ## Changelog
74
91
 
75
92
  ### Version 3.0
@@ -6,6 +6,7 @@ import itertools
6
6
  from typing import List, Tuple
7
7
  from scholarly import scholarly, ProxyGenerator
8
8
  from geopy.geocoders import Nominatim
9
+
9
10
  import folium
10
11
  from tqdm import tqdm
11
12
  from multiprocessing import Pool
@@ -19,35 +20,55 @@ def fetch_google_scholar_profile(scholar_id: str):
19
20
  author = scholarly.fill(author, sections=['publications'])
20
21
  return author
21
22
 
22
- def find_all_citing_institutions(publications, num_processes: int = 16) -> List[Tuple[str]]:
23
+ def find_all_citing_institutions(publications, num_processes_major: int = 16, num_processes_minor: int = 32) -> List[Tuple[str]]:
23
24
  '''
24
25
  Step 2. Final all citing institutions (i.e., insitutions of citing authors).
25
26
  '''
26
- # Fetch metadata for all publications, in parallel.
27
- with Pool(processes=num_processes) as pool:
28
- all_publications = list(tqdm(pool.imap(__fill_publication_metadata, publications),
29
- desc='Filling metadata for all publications...',
30
- total=len(publications)))
31
-
32
- # Find all citing papers from all publications, in parallel.
33
- with Pool(processes=num_processes) as pool:
34
- all_citing_papers_nested = list(tqdm(pool.imap(__citing_papers_from_publication, all_publications),
35
- desc='Finding all citations...',
36
- total=len(all_publications)))
27
+ if num_processes_minor > 1 and isinstance(num_processes_minor, int):
28
+ # Fetch metadata for all publications, in parallel.
29
+ with Pool(processes=num_processes_minor) as pool:
30
+ all_publications = list(tqdm(pool.imap(__fill_publication_metadata, publications),
31
+ desc='Filling metadata for all publications...',
32
+ total=len(publications)))
33
+ else:
34
+ all_publications = []
35
+ for pub in tqdm(publications, desc='Filling metadata for all publications...', total=len(publications)):
36
+ all_publications.append(__fill_publication_metadata(pub))
37
+
38
+ if num_processes_major > 1 and isinstance(num_processes_major, int):
39
+ # Find all citing papers from all publications, in parallel.
40
+ with Pool(processes=num_processes_major) as pool:
41
+ all_citing_papers_nested = list(tqdm(pool.imap(__citing_papers_from_publication, all_publications),
42
+ desc='Finding all citations...',
43
+ total=len(all_publications)))
44
+ else:
45
+ all_citing_papers_nested = []
46
+ for pub in tqdm(all_publications, desc='Finding all citations...', total=len(all_publications)):
47
+ all_citing_papers_nested.append(__citing_papers_from_publication_difficult(pub))
37
48
  all_citing_papers = list(itertools.chain(*all_citing_papers_nested))
38
49
 
39
- # Find all citing authors from all citing papers, in parallel.
40
- with Pool(processes=num_processes) as pool:
41
- all_citing_authors_nested = list(tqdm(pool.imap(__authors_from_papers, all_citing_papers),
42
- desc='Finding all citing authors...',
43
- total=len(all_citing_papers)))
50
+ if num_processes_minor > 1 and isinstance(num_processes_minor, int):
51
+ # Find all citing authors from all citing papers, in parallel.
52
+ with Pool(processes=num_processes_minor) as pool:
53
+ all_citing_authors_nested = list(tqdm(pool.imap(__authors_from_papers, all_citing_papers),
54
+ desc='Finding all citing authors...',
55
+ total=len(all_citing_papers)))
56
+ else:
57
+ all_citing_authors_nested = []
58
+ for paper in tqdm(all_citing_papers, desc='Finding all citing authors...', total=len(all_citing_papers)):
59
+ all_citing_authors_nested.append(__authors_from_papers(paper))
44
60
  all_citing_authors = list(itertools.chain(*all_citing_authors_nested))
45
61
 
46
- # Find all citing insitutions from all citing authors, in parallel.
47
- with Pool(processes=num_processes) as pool:
48
- author_institution_tuple_list = list(tqdm(pool.imap(__institutions_from_authors, all_citing_authors),
49
- desc='Finding all citing institutions...',
50
- total=len(all_citing_authors)))
62
+ if num_processes_minor > 1 and isinstance(num_processes_minor, int):
63
+ # Find all citing insitutions from all citing authors, in parallel.
64
+ with Pool(processes=num_processes_minor) as pool:
65
+ author_institution_tuple_list = list(tqdm(pool.imap(__institutions_from_authors, all_citing_authors),
66
+ desc='Finding all citing institutions...',
67
+ total=len(all_citing_authors)))
68
+ else:
69
+ author_institution_tuple_list = []
70
+ for author in tqdm(all_citing_authors, desc='Finding all citing institutions...', total=len(all_citing_authors)):
71
+ author_institution_tuple_list.append(__institutions_from_authors(author))
51
72
 
52
73
  return author_institution_tuple_list
53
74
 
@@ -72,14 +93,22 @@ def clean_institution_names(author_institution_tuple_list: List[Tuple[str]]) ->
72
93
  cleaned_author_institution_tuple_list.append((author_name, cleaned_institution))
73
94
  return cleaned_author_institution_tuple_list
74
95
 
75
- def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]], num_processes: int = 16) -> List[Tuple[str]]:
96
+ def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]], num_processes_minor: int = 32) -> List[Tuple[str]]:
76
97
  '''
77
98
  Step 4: Convert institutions in plain text to Geocode.
78
99
  '''
79
- with Pool(processes=num_processes) as pool:
80
- coordinates = list(tqdm(pool.imap(__text_to_geocode, author_institution_tuple_list),
81
- desc='Finding all geographic coordinates...',
82
- total=len(author_institution_tuple_list)))
100
+ if num_processes_minor > 1 and isinstance(num_processes_minor, int):
101
+ with Pool(processes=num_processes_minor) as pool:
102
+ coordinates = list(tqdm(pool.imap(__text_to_geocode, author_institution_tuple_list),
103
+ desc='Finding all geographic coordinates...',
104
+ total=len(author_institution_tuple_list)))
105
+ else:
106
+ coordinates = []
107
+ for author_institution_tuple in tqdm(author_institution_tuple_list,
108
+ desc='Finding all geographic coordinates...',
109
+ total=len(author_institution_tuple_list)):
110
+ coordinates.append(__text_to_geocode(author_institution_tuple))
111
+
83
112
  coordinates = [item for item in coordinates if item is not None] # Filter out empty coordinates.
84
113
  return coordinates
85
114
 
@@ -108,8 +137,22 @@ def __fill_publication_metadata(pub):
108
137
  def __citing_papers_from_publication(pub):
109
138
  if 'citedby_url' in pub:
110
139
  time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
111
- citing_paper_obj = scholarly.citedby(pub)
112
- return [item for item in citing_paper_obj]
140
+ citing_paper_iterator = scholarly.citedby(pub)
141
+ citing_paper_list = [item for item in citing_paper_iterator]
142
+ return citing_paper_list
143
+ return []
144
+
145
+ def __citing_papers_from_publication_difficult(pub):
146
+ if 'citedby_url' in pub:
147
+ time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
148
+ citing_paper_iterator = scholarly.citedby(pub)
149
+ citing_paper_list = []
150
+ # Unpacking the iterator to a list invokes `scholarly`,
151
+ # which can easily get us blocked if not handled carefully.
152
+ for item in citing_paper_iterator:
153
+ time.sleep(random.uniform(1, 2)) # Random delay to reduce risk of being blocked.
154
+ citing_paper_list.append(item)
155
+ return citing_paper_list
113
156
  return []
114
157
 
115
158
  def __authors_from_papers(citing_paper):
@@ -127,19 +170,22 @@ def __institutions_from_authors(author_id):
127
170
  return (citing_author['name'], citing_author['affiliation'])
128
171
  return []
129
172
 
130
- def __text_to_geocode(author_institution_tuple):
173
+ def __text_to_geocode(author_institution_tuple, max_attempts: int = 2):
131
174
  geolocator = Nominatim(user_agent='citation_mapper')
132
- try:
133
- author_name, institution_name = author_institution_tuple
134
- geo_location = geolocator.geocode(institution_name)
135
- if geo_location:
136
- return (geo_location.latitude, geo_location.longitude, author_name, institution_name)
137
- except:
138
- return None
175
+ for _ in range(max_attempts):
176
+ try:
177
+ author_name, institution_name = author_institution_tuple
178
+ geo_location = geolocator.geocode(institution_name)
179
+ if geo_location:
180
+ return (geo_location.latitude, geo_location.longitude, author_name, institution_name)
181
+ except:
182
+ pass
183
+ return None
139
184
 
140
185
  def generate_citation_map(scholar_id: str,
141
186
  output_path: str = 'citation_map.html',
142
- num_processes: int = 16,
187
+ num_processes_major: int = 16,
188
+ num_processes_minor: int = 32,
143
189
  use_proxy: bool = False,
144
190
  pin_colorful: bool = True,
145
191
  print_citing_institutions: bool = True):
@@ -153,9 +199,14 @@ def generate_citation_map(scholar_id: str,
153
199
  output_path: str
154
200
  (default is 'citation_map.html')
155
201
  The path to the output HTML file.
156
- num_processes: int
202
+ num_processes_major: int
157
203
  (default is 16)
158
- Number of separate processes for parallel processing.
204
+ Number of processes for parallel processing.
205
+ Related to `scholarly.citedby()`. This causes Google Scholar blocks if set too high.
206
+ num_processes_minor: int
207
+ (default is 32)
208
+ Number of processes for parallel processing.
209
+ Related to all other processes. Usually not causing problems.
159
210
  use_proxy: bool
160
211
  (default is False)
161
212
  If true, we will use a scholarly proxy.
@@ -178,7 +229,9 @@ def generate_citation_map(scholar_id: str,
178
229
  author_profile = fetch_google_scholar_profile(scholar_id)
179
230
  print('\nAuthor profile found, with %d publications.\n' % len(author_profile['publications']))
180
231
 
181
- author_institution_tuple_list = find_all_citing_institutions(author_profile['publications'], num_processes=num_processes)
232
+ author_institution_tuple_list = find_all_citing_institutions(author_profile['publications'],
233
+ num_processes_major=num_processes_major,
234
+ num_processes_minor=num_processes_minor)
182
235
  print('\nA total of %d citing institutions recorded.\n' % len(author_institution_tuple_list))
183
236
 
184
237
  if print_citing_institutions:
@@ -193,8 +246,9 @@ def generate_citation_map(scholar_id: str,
193
246
  for item in sorted(cleaned_author_institution_tuple_list):
194
247
  print(item)
195
248
 
196
- coordinates = institution_text_to_geocode(author_institution_tuple_list + cleaned_author_institution_tuple_list)
197
- print('\nConverted the institutions to Geocodes.')
249
+ coordinates = institution_text_to_geocode(author_institution_tuple_list + cleaned_author_institution_tuple_list,
250
+ num_processes_minor=num_processes_minor)
251
+ print('\nConverted the institutions to %d Geocodes.' % len(coordinates))
198
252
 
199
253
  citation_map = create_map(coordinates, pin_colorful=pin_colorful)
200
254
  citation_map.save(output_path)
@@ -206,4 +260,4 @@ if __name__ == '__main__':
206
260
  # Replace this with your Google Scholar ID.
207
261
  scholar_id = '3rDjnykAAAAJ'
208
262
  generate_citation_map(scholar_id, output_path='citation_map.html',
209
- num_processes=16, use_proxy=False, pin_colorful=True, print_citing_institutions=True)
263
+ num_processes_minor=16, use_proxy=False, pin_colorful=True, print_citing_institutions=True)
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: citation-map
3
- Version: 3.3
3
+ Version: 3.5
4
4
  Summary: A simple tool to generate your Google Scholar citation world map.
5
5
  Home-page: https://github.com/ChenLiu-1996/CitationMap
6
6
  Author: Chen Liu
@@ -17,7 +17,6 @@ License-File: LICENSE.md
17
17
  # Google Scholar Citation World Map
18
18
 
19
19
  [![Latest PyPI version](https://img.shields.io/pypi/v/citation-map.svg)](https://pypi.org/project/citation-map/)
20
- [![PyPI license](https://img.shields.io/pypi/l/citation-map.svg)](https://pypi.python.org/pypi/citation-map/)
21
20
  [![PyPI download month](https://img.shields.io/pypi/dm/citation-map.svg)](https://pypi.python.org/pypi/citation-map/)
22
21
  [![PyPI download day](https://img.shields.io/pypi/dd/citation-map.svg)](https://pypi.python.org/pypi/citation-map/)
23
22
  [![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
@@ -32,6 +31,8 @@ This is a simple Python tool to generate an HTML citation world map from your Go
32
31
 
33
32
  It is easy to install (`pip install citation-map`, available on [PyPI](https://pypi.org/project/citation-map/)) and easy to use (see the [Usage](https://github.com/ChenLiu-1996/CitationMap?tab=readme-ov-file#usage) section).
34
33
 
34
+ **You don't need to fork this repo unless you want to make custom changes.**
35
+
35
36
  ## Expected Outcome
36
37
  You will be given an HTML file as the output of the script.
37
38
 
@@ -45,6 +46,10 @@ If you open it on a browser, you will see your own version of the following cita
45
46
  ```
46
47
  pip install citation-map
47
48
  ```
49
+ Remember to upgrade to the latest version if you installed an earlier version.
50
+ ```
51
+ pip install citation-map --upgrade
52
+ ```
48
53
 
49
54
  2. Find your Google Scholar ID.
50
55
 
@@ -70,9 +75,14 @@ If you open it on a browser, you will see your own version of the following cita
70
75
  output_path: str
71
76
  (default is 'citation_map.html')
72
77
  The path to the output HTML file.
73
- num_processes: int
78
+ num_processes_major: int
74
79
  (default is 16)
75
- Number of separate processes for parallel processing.
80
+ Number of processes for parallel processing.
81
+ Related to `scholarly.citedby()`. This causes Google Scholar blocks if set too high.
82
+ num_processes_minor: int
83
+ (default is 32)
84
+ Number of processes for parallel processing.
85
+ Related to all other processes. Usually not causing problems.
76
86
  use_proxy: bool
77
87
  (default is False)
78
88
  If true, we will use a scholarly proxy.
@@ -86,6 +96,13 @@ If you open it on a browser, you will see your own version of the following cita
86
96
  If true, print the list of citing institutions (affiliations of citing authors).
87
97
  ```
88
98
 
99
+ ## Debug
100
+ 1. `scholarly._proxy_generator.MaxTriesExceededException`
101
+
102
+ - From my experience, this is a good indicator that your IP address is blocked by Google Scholar due to excessive or frequent crawling (using the `scholarly` package).
103
+ - One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
104
+ - In case this does not help, you can try to reduce the number of processes (e.g., setting `num_processes_major=1`).
105
+
89
106
  ## Changelog
90
107
 
91
108
  ### Version 3.0
@@ -5,7 +5,7 @@ with open('README.md') as f:
5
5
 
6
6
  setup(
7
7
  name='citation-map',
8
- version='3.3',
8
+ version='3.5',
9
9
  license='Apache 2.0',
10
10
  author='Chen Liu',
11
11
  author_email='chen.liu.cl2482@yale.edu',
File without changes
File without changes