citation-map 3.6__tar.gz → 3.8__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {citation-map-3.6 → citation-map-3.8}/PKG-INFO +16 -12
- {citation-map-3.6 → citation-map-3.8}/README.md +15 -11
- citation-map-3.8/citation_map/__init__.py +5 -0
- {citation-map-3.6 → citation-map-3.8}/citation_map/citation_map.py +71 -102
- citation-map-3.8/citation_map/scholarly_support.py +92 -0
- {citation-map-3.6 → citation-map-3.8}/citation_map.egg-info/PKG-INFO +16 -12
- {citation-map-3.6 → citation-map-3.8}/citation_map.egg-info/SOURCES.txt +1 -0
- {citation-map-3.6 → citation-map-3.8}/citation_map.egg-info/requires.txt +2 -0
- {citation-map-3.6 → citation-map-3.8}/setup.py +2 -2
- citation-map-3.6/citation_map/__init__.py +0 -4
- {citation-map-3.6 → citation-map-3.8}/LICENSE.md +0 -0
- {citation-map-3.6 → citation-map-3.8}/citation_map.egg-info/dependency_links.txt +0 -0
- {citation-map-3.6 → citation-map-3.8}/citation_map.egg-info/top_level.txt +0 -0
- {citation-map-3.6 → citation-map-3.8}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: citation-map
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.8
|
|
4
4
|
Summary: A simple tool to generate your Google Scholar citation world map.
|
|
5
5
|
Home-page: https://github.com/ChenLiu-1996/CitationMap
|
|
6
6
|
Author: Chen Liu
|
|
@@ -75,14 +75,9 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
75
75
|
output_path: str
|
|
76
76
|
(default is 'citation_map.html')
|
|
77
77
|
The path to the output HTML file.
|
|
78
|
-
|
|
78
|
+
num_processes: int
|
|
79
79
|
(default is 16)
|
|
80
80
|
Number of processes for parallel processing.
|
|
81
|
-
Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
|
|
82
|
-
num_processes_minor: int
|
|
83
|
-
(default is 32)
|
|
84
|
-
Number of processes for parallel processing.
|
|
85
|
-
Related to all other processes. Usually not causing problems.
|
|
86
81
|
use_proxy: bool
|
|
87
82
|
(default is False)
|
|
88
83
|
If true, we will use a scholarly proxy.
|
|
@@ -97,18 +92,27 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
97
92
|
```
|
|
98
93
|
|
|
99
94
|
## Debug
|
|
100
|
-
1. `scholarly._proxy_generator.MaxTriesExceededException`
|
|
95
|
+
1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
|
|
101
96
|
|
|
102
|
-
- From my experience,
|
|
97
|
+
- From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
|
|
103
98
|
- One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
|
|
104
|
-
- In case this does not help, you can try to reduce the number of processes (e.g., setting `
|
|
99
|
+
- In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
|
|
105
100
|
|
|
106
101
|
## Changelog
|
|
107
102
|
|
|
103
|
+
### Version 3.7
|
|
104
|
+
I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
|
|
105
|
+
|
|
106
|
+
**Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
|
|
107
|
+
|
|
108
108
|
### Version 3.0
|
|
109
|
-
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
109
|
+
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
110
|
+
|
|
111
|
+
For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
|
|
112
|
+
|
|
113
|
+
I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
|
|
110
114
|
|
|
111
|
-
|
|
115
|
+
**Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
|
|
112
116
|
|
|
113
117
|
### Version 2.0
|
|
114
118
|
I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
|
|
@@ -59,14 +59,9 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
59
59
|
output_path: str
|
|
60
60
|
(default is 'citation_map.html')
|
|
61
61
|
The path to the output HTML file.
|
|
62
|
-
|
|
62
|
+
num_processes: int
|
|
63
63
|
(default is 16)
|
|
64
64
|
Number of processes for parallel processing.
|
|
65
|
-
Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
|
|
66
|
-
num_processes_minor: int
|
|
67
|
-
(default is 32)
|
|
68
|
-
Number of processes for parallel processing.
|
|
69
|
-
Related to all other processes. Usually not causing problems.
|
|
70
65
|
use_proxy: bool
|
|
71
66
|
(default is False)
|
|
72
67
|
If true, we will use a scholarly proxy.
|
|
@@ -81,18 +76,27 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
81
76
|
```
|
|
82
77
|
|
|
83
78
|
## Debug
|
|
84
|
-
1. `scholarly._proxy_generator.MaxTriesExceededException`
|
|
79
|
+
1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
|
|
85
80
|
|
|
86
|
-
- From my experience,
|
|
81
|
+
- From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
|
|
87
82
|
- One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
|
|
88
|
-
- In case this does not help, you can try to reduce the number of processes (e.g., setting `
|
|
83
|
+
- In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
|
|
89
84
|
|
|
90
85
|
## Changelog
|
|
91
86
|
|
|
87
|
+
### Version 3.7
|
|
88
|
+
I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
|
|
89
|
+
|
|
90
|
+
**Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
|
|
91
|
+
|
|
92
92
|
### Version 3.0
|
|
93
|
-
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
93
|
+
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
94
|
+
|
|
95
|
+
For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
|
|
96
|
+
|
|
97
|
+
I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
|
|
94
98
|
|
|
95
|
-
|
|
99
|
+
**Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
|
|
96
100
|
|
|
97
101
|
### Version 2.0
|
|
98
102
|
I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
|
|
@@ -11,6 +11,8 @@ import folium
|
|
|
11
11
|
from tqdm import tqdm
|
|
12
12
|
from multiprocessing import Pool
|
|
13
13
|
|
|
14
|
+
from .scholarly_support import get_author_ids
|
|
15
|
+
|
|
14
16
|
|
|
15
17
|
def fetch_google_scholar_profile(scholar_id: str):
|
|
16
18
|
'''
|
|
@@ -20,54 +22,50 @@ def fetch_google_scholar_profile(scholar_id: str):
|
|
|
20
22
|
author = scholarly.fill(author, sections=['publications'])
|
|
21
23
|
return author
|
|
22
24
|
|
|
23
|
-
def find_all_citing_institutions(publications,
|
|
25
|
+
def find_all_citing_institutions(publications, num_processes: int = 16) -> List[Tuple[str]]:
|
|
24
26
|
'''
|
|
25
|
-
Step 2.
|
|
27
|
+
Step 2. Find all citing institutions (i.e., insitutions of citing authors).
|
|
26
28
|
'''
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
with Pool(processes=
|
|
29
|
+
# Fetch metadata for all publications.
|
|
30
|
+
if num_processes > 1 and isinstance(num_processes, int):
|
|
31
|
+
with Pool(processes=num_processes) as pool:
|
|
30
32
|
all_publications = list(tqdm(pool.imap(__fill_publication_metadata, publications),
|
|
31
|
-
desc='Filling metadata for all publications
|
|
33
|
+
desc='Filling metadata for all publications',
|
|
32
34
|
total=len(publications)))
|
|
33
35
|
else:
|
|
34
36
|
all_publications = []
|
|
35
|
-
for pub in tqdm(publications, desc='Filling metadata for all publications
|
|
37
|
+
for pub in tqdm(publications, desc='Filling metadata for all publications', total=len(publications)):
|
|
36
38
|
all_publications.append(__fill_publication_metadata(pub))
|
|
37
39
|
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
with Pool(processes=num_processes_minor) as pool:
|
|
53
|
-
all_citing_authors_nested = list(tqdm(pool.imap(__authors_from_papers, all_citing_papers),
|
|
54
|
-
desc='Finding all citing authors...',
|
|
55
|
-
total=len(all_citing_papers)))
|
|
40
|
+
# Convert all publications to Google Scholar publication IDs.
|
|
41
|
+
# This is fast and no parallel processing is needed.
|
|
42
|
+
all_publication_ids = []
|
|
43
|
+
for pub in all_publications:
|
|
44
|
+
if 'cites_id' in pub:
|
|
45
|
+
for cites_id in pub['cites_id']:
|
|
46
|
+
all_publication_ids.append(cites_id)
|
|
47
|
+
|
|
48
|
+
# Find all citing authors from all publications.
|
|
49
|
+
if num_processes > 1 and isinstance(num_processes, int):
|
|
50
|
+
with Pool(processes=num_processes) as pool:
|
|
51
|
+
all_citing_authors_nested = list(tqdm(pool.imap(__citing_author_ids_from_publication, all_publication_ids),
|
|
52
|
+
desc='Finding all citing authors from cited publications',
|
|
53
|
+
total=len(all_publication_ids)))
|
|
56
54
|
else:
|
|
57
55
|
all_citing_authors_nested = []
|
|
58
|
-
for
|
|
59
|
-
all_citing_authors_nested.append(
|
|
56
|
+
for pub in tqdm(all_publication_ids, desc='Finding all citing authors from cited publications', total=len(all_publication_ids)):
|
|
57
|
+
all_citing_authors_nested.append(__citing_author_ids_from_publication(pub))
|
|
60
58
|
all_citing_authors = list(itertools.chain(*all_citing_authors_nested))
|
|
61
59
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
with Pool(processes=
|
|
60
|
+
# Find all citing insitutions from all citing authors.
|
|
61
|
+
if num_processes > 1 and isinstance(num_processes, int):
|
|
62
|
+
with Pool(processes=num_processes) as pool:
|
|
65
63
|
author_institution_tuple_list = list(tqdm(pool.imap(__institutions_from_authors, all_citing_authors),
|
|
66
|
-
desc='Finding all citing institutions
|
|
64
|
+
desc='Finding all citing institutions from citing authors',
|
|
67
65
|
total=len(all_citing_authors)))
|
|
68
66
|
else:
|
|
69
67
|
author_institution_tuple_list = []
|
|
70
|
-
for author in tqdm(all_citing_authors, desc='Finding all citing institutions
|
|
68
|
+
for author in tqdm(all_citing_authors, desc='Finding all citing institutions from citing authors', total=len(all_citing_authors)):
|
|
71
69
|
author_institution_tuple_list.append(__institutions_from_authors(author))
|
|
72
70
|
|
|
73
71
|
return author_institution_tuple_list
|
|
@@ -75,6 +73,8 @@ def find_all_citing_institutions(publications, num_processes_major: int = 16, nu
|
|
|
75
73
|
def clean_institution_names(author_institution_tuple_list: List[Tuple[str]]) -> List[Tuple[str]]:
|
|
76
74
|
'''
|
|
77
75
|
Step 3. Clean up the name of institutions from the author's self-entered Google Scholar affiliation.
|
|
76
|
+
NOTE: This logic is very naive. Please send an issue or pull request if you have any idea how to improve it.
|
|
77
|
+
Currently we will not consider any paid service or tools that pose extra burden on the users, such as GPT API.
|
|
78
78
|
'''
|
|
79
79
|
cleaned_author_institution_tuple_list = []
|
|
80
80
|
for author_name, institution_string in author_institution_tuple_list:
|
|
@@ -83,32 +83,37 @@ def clean_institution_names(author_institution_tuple_list: List[Tuple[str]]) ->
|
|
|
83
83
|
for substring in substring_list:
|
|
84
84
|
# Use a regular expression to remove anything before 'at', or '@'.
|
|
85
85
|
cleaned_institution = re.sub(r'.*?\bat\b|.*?@', '', substring, flags=re.IGNORECASE).strip()
|
|
86
|
-
# Use a regular expression to
|
|
87
|
-
|
|
86
|
+
# Use a regular expression to filter out strings that represent
|
|
87
|
+
# a person's identity rather than affiliation.
|
|
88
|
+
is_common_identity_string = re.search(
|
|
88
89
|
re.compile(
|
|
89
90
|
r'\b(director|manager|chair|engineer|programmer|scientist|professor|lecturer|phd|student|ph\.d|department of)\b',
|
|
90
91
|
re.IGNORECASE),
|
|
91
92
|
substring)
|
|
92
|
-
if not
|
|
93
|
+
if not is_common_identity_string:
|
|
93
94
|
cleaned_author_institution_tuple_list.append((author_name, cleaned_institution))
|
|
94
95
|
return cleaned_author_institution_tuple_list
|
|
95
96
|
|
|
96
|
-
def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]],
|
|
97
|
+
def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]], max_attempts: int = 3) -> List[Tuple[str]]:
|
|
97
98
|
'''
|
|
98
99
|
Step 4: Convert institutions in plain text to Geocode.
|
|
99
100
|
'''
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
101
|
+
coordinates = []
|
|
102
|
+
# NOTE: According to the Nomatim Usage Policy (https://operations.osmfoundation.org/policies/nominatim/),
|
|
103
|
+
# we are explicitly asked not to submit bulk requests on multiple threads.
|
|
104
|
+
# Therefore, we will keep it to a loop instead of multiprocessing.
|
|
105
|
+
geolocator = Nominatim(user_agent='citation_mapper')
|
|
106
|
+
for author_institution_tuple in tqdm(author_institution_tuple_list,
|
|
107
|
+
desc='Finding all geographic coordinates from citing institutions',
|
|
108
|
+
total=len(author_institution_tuple_list)):
|
|
109
|
+
for _ in range(max_attempts):
|
|
110
|
+
try:
|
|
111
|
+
author_name, institution_name = author_institution_tuple
|
|
112
|
+
geo_location = geolocator.geocode(institution_name)
|
|
113
|
+
if geo_location:
|
|
114
|
+
coordinates.append((geo_location.latitude, geo_location.longitude, author_name, institution_name))
|
|
115
|
+
except:
|
|
116
|
+
continue
|
|
112
117
|
coordinates = [item for item in coordinates if item is not None] # Filter out empty coordinates.
|
|
113
118
|
return coordinates
|
|
114
119
|
|
|
@@ -134,58 +139,23 @@ def __fill_publication_metadata(pub):
|
|
|
134
139
|
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
135
140
|
return scholarly.fill(pub)
|
|
136
141
|
|
|
137
|
-
def
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
citing_paper_list = [item for item in citing_paper_iterator]
|
|
142
|
-
return citing_paper_list
|
|
143
|
-
return []
|
|
142
|
+
def __citing_author_ids_from_publication(cites_id: str):
|
|
143
|
+
citing_paper_search_url = 'https://scholar.google.com/scholar?hl=en&cites=' + cites_id
|
|
144
|
+
citing_author_ids = get_author_ids(citing_paper_search_url)
|
|
145
|
+
return citing_author_ids
|
|
144
146
|
|
|
145
|
-
def
|
|
146
|
-
if 'citedby_url' in pub:
|
|
147
|
-
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
148
|
-
citing_paper_iterator = scholarly.citedby(pub)
|
|
149
|
-
citing_paper_list = []
|
|
150
|
-
# Unpacking the iterator to a list invokes `scholarly`,
|
|
151
|
-
# which can easily get us blocked if not handled carefully.
|
|
152
|
-
for item in citing_paper_iterator:
|
|
153
|
-
time.sleep(random.uniform(1, 2)) # Random delay to reduce risk of being blocked.
|
|
154
|
-
citing_paper_list.append(item)
|
|
155
|
-
return citing_paper_list
|
|
156
|
-
return []
|
|
157
|
-
|
|
158
|
-
def __authors_from_papers(citing_paper):
|
|
159
|
-
if 'author_id' in citing_paper:
|
|
160
|
-
author_id_list = citing_paper['author_id']
|
|
161
|
-
author_id_list = [item for item in author_id_list if item] # Filter out empty author ids.
|
|
162
|
-
return author_id_list
|
|
163
|
-
return []
|
|
164
|
-
|
|
165
|
-
def __institutions_from_authors(author_id):
|
|
147
|
+
def __institutions_from_authors(author_id: str):
|
|
166
148
|
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
167
149
|
citing_author = scholarly.search_author_id(author_id)
|
|
168
|
-
citing_author = scholarly.fill(citing_author)
|
|
150
|
+
# citing_author = scholarly.fill(citing_author)
|
|
169
151
|
if 'affiliation' in citing_author:
|
|
170
152
|
return (citing_author['name'], citing_author['affiliation'])
|
|
171
153
|
return []
|
|
172
154
|
|
|
173
|
-
def __text_to_geocode(author_institution_tuple, max_attempts: int = 2):
|
|
174
|
-
geolocator = Nominatim(user_agent='citation_mapper')
|
|
175
|
-
for _ in range(max_attempts):
|
|
176
|
-
try:
|
|
177
|
-
author_name, institution_name = author_institution_tuple
|
|
178
|
-
geo_location = geolocator.geocode(institution_name)
|
|
179
|
-
if geo_location:
|
|
180
|
-
return (geo_location.latitude, geo_location.longitude, author_name, institution_name)
|
|
181
|
-
except:
|
|
182
|
-
pass
|
|
183
|
-
return None
|
|
184
155
|
|
|
185
156
|
def generate_citation_map(scholar_id: str,
|
|
186
157
|
output_path: str = 'citation_map.html',
|
|
187
|
-
|
|
188
|
-
num_processes_minor: int = 32,
|
|
158
|
+
num_processes: int = 16,
|
|
189
159
|
use_proxy: bool = False,
|
|
190
160
|
pin_colorful: bool = True,
|
|
191
161
|
print_citing_institutions: bool = True):
|
|
@@ -199,14 +169,9 @@ def generate_citation_map(scholar_id: str,
|
|
|
199
169
|
output_path: str
|
|
200
170
|
(default is 'citation_map.html')
|
|
201
171
|
The path to the output HTML file.
|
|
202
|
-
|
|
172
|
+
num_processes: int
|
|
203
173
|
(default is 16)
|
|
204
174
|
Number of processes for parallel processing.
|
|
205
|
-
Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
|
|
206
|
-
num_processes_minor: int
|
|
207
|
-
(default is 32)
|
|
208
|
-
Number of processes for parallel processing.
|
|
209
|
-
Related to all other processes. Usually not causing problems.
|
|
210
175
|
use_proxy: bool
|
|
211
176
|
(default is False)
|
|
212
177
|
If true, we will use a scholarly proxy.
|
|
@@ -230,24 +195,28 @@ def generate_citation_map(scholar_id: str,
|
|
|
230
195
|
print('\nAuthor profile found, with %d publications.\n' % len(author_profile['publications']))
|
|
231
196
|
|
|
232
197
|
author_institution_tuple_list = find_all_citing_institutions(author_profile['publications'],
|
|
233
|
-
|
|
234
|
-
num_processes_minor=num_processes_minor)
|
|
198
|
+
num_processes=num_processes)
|
|
235
199
|
print('\nA total of %d citing institutions recorded.\n' % len(author_institution_tuple_list))
|
|
236
200
|
|
|
201
|
+
# Take unique tuples.
|
|
202
|
+
author_institution_tuple_list = list(set(author_institution_tuple_list))
|
|
203
|
+
|
|
237
204
|
if print_citing_institutions:
|
|
238
205
|
print('\nList of all citing authors and institutions before cleaning:\n')
|
|
239
|
-
for item in sorted(author_institution_tuple_list):
|
|
206
|
+
for item in sorted((author_institution_tuple_list)):
|
|
240
207
|
print(item)
|
|
241
208
|
|
|
242
209
|
cleaned_author_institution_tuple_list = clean_institution_names(author_institution_tuple_list)
|
|
243
210
|
|
|
211
|
+
# Take unique tuples.
|
|
212
|
+
cleaned_author_institution_tuple_list = list(set(cleaned_author_institution_tuple_list))
|
|
213
|
+
|
|
244
214
|
if print_citing_institutions:
|
|
245
215
|
print('\nList of all citing authors and institutions after cleaning:\n')
|
|
246
216
|
for item in sorted(cleaned_author_institution_tuple_list):
|
|
247
217
|
print(item)
|
|
248
218
|
|
|
249
|
-
coordinates = institution_text_to_geocode(author_institution_tuple_list + cleaned_author_institution_tuple_list
|
|
250
|
-
num_processes_minor=num_processes_minor)
|
|
219
|
+
coordinates = institution_text_to_geocode(author_institution_tuple_list + cleaned_author_institution_tuple_list)
|
|
251
220
|
print('\nConverted the institutions to %d Geocodes.' % len(coordinates))
|
|
252
221
|
|
|
253
222
|
citation_map = create_map(coordinates, pin_colorful=pin_colorful)
|
|
@@ -260,4 +229,4 @@ if __name__ == '__main__':
|
|
|
260
229
|
# Replace this with your Google Scholar ID.
|
|
261
230
|
scholar_id = '3rDjnykAAAAJ'
|
|
262
231
|
generate_citation_map(scholar_id, output_path='citation_map.html',
|
|
263
|
-
|
|
232
|
+
num_processes=16, use_proxy=False, pin_colorful=True, print_citing_institutions=True)
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
import requests
|
|
2
|
+
from bs4 import BeautifulSoup
|
|
3
|
+
from typing import List
|
|
4
|
+
import time
|
|
5
|
+
import random
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
def get_author_ids(paper_url: str) -> List[str]:
|
|
9
|
+
'''
|
|
10
|
+
Find the Google Scholar IDs of authors from a Google Scholar paper url.
|
|
11
|
+
'''
|
|
12
|
+
|
|
13
|
+
citing_author_ids = []
|
|
14
|
+
|
|
15
|
+
headers = requests.utils.default_headers()
|
|
16
|
+
headers.update({
|
|
17
|
+
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
|
|
18
|
+
})
|
|
19
|
+
|
|
20
|
+
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
21
|
+
|
|
22
|
+
# Search the url of all citing papers.
|
|
23
|
+
response = requests.get(paper_url, headers=headers)
|
|
24
|
+
if response.status_code != 200:
|
|
25
|
+
raise Exception('Failed to fetch the Google Scholar page')
|
|
26
|
+
|
|
27
|
+
# Get the HTML data.
|
|
28
|
+
soup = BeautifulSoup(response.text, 'html.parser')
|
|
29
|
+
|
|
30
|
+
# Check for common indicators of blocking
|
|
31
|
+
if 'CAPTCHA' in soup.text or 'not a robot' in soup.text:
|
|
32
|
+
print('[WARNING!] Blocked by CAPTCHA or robot check when searching %s.' % paper_url)
|
|
33
|
+
return []
|
|
34
|
+
|
|
35
|
+
if 'Access Denied' in soup.text or 'Forbidden' in soup.text:
|
|
36
|
+
print('[WARNING!] Access denied or forbidden when searching searching %s.' % paper_url)
|
|
37
|
+
return []
|
|
38
|
+
|
|
39
|
+
# Loop through the citation results
|
|
40
|
+
current_page_number = 1
|
|
41
|
+
while True:
|
|
42
|
+
for result in soup.find_all('div', class_='gs_ri'):
|
|
43
|
+
title_tag = result.find('h3', class_='gs_rt')
|
|
44
|
+
if title_tag:
|
|
45
|
+
author_links = result.find_all('a', href=True)
|
|
46
|
+
for link in author_links:
|
|
47
|
+
if 'user=' in link['href']:
|
|
48
|
+
author_id = link['href'].split('user=')[1].split('&')[0]
|
|
49
|
+
citing_author_ids.append(author_id)
|
|
50
|
+
else:
|
|
51
|
+
continue
|
|
52
|
+
|
|
53
|
+
# Find the page navigation.
|
|
54
|
+
navigation_buttons = soup.find_all('a', class_='gs_nma')
|
|
55
|
+
for navigation in navigation_buttons:
|
|
56
|
+
page_number_str = navigation.text
|
|
57
|
+
if page_number_str and page_number_str.isnumeric() and int(page_number_str) == current_page_number + 1:
|
|
58
|
+
# Found the correct button for next page.
|
|
59
|
+
current_page_number += 1
|
|
60
|
+
next_url = 'https://scholar.google.com' + navigation['href']
|
|
61
|
+
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
62
|
+
response = requests.get(next_url, headers=headers)
|
|
63
|
+
if response.status_code != 200:
|
|
64
|
+
break
|
|
65
|
+
soup = BeautifulSoup(response.text, 'html.parser')
|
|
66
|
+
else:
|
|
67
|
+
continue
|
|
68
|
+
else:
|
|
69
|
+
break
|
|
70
|
+
|
|
71
|
+
return citing_author_ids
|
|
72
|
+
|
|
73
|
+
def scrape_citations(citation_url, headers):
|
|
74
|
+
response = requests.get(citation_url, headers=headers)
|
|
75
|
+
if response.status_code != 200:
|
|
76
|
+
raise Exception('Failed to fetch the citations page')
|
|
77
|
+
|
|
78
|
+
soup = BeautifulSoup(response.text, 'html.parser')
|
|
79
|
+
|
|
80
|
+
papers = []
|
|
81
|
+
|
|
82
|
+
for result in soup.find_all('div', class_='gs_ri'):
|
|
83
|
+
title = result.find('h3', class_='gs_rt').text
|
|
84
|
+
authors = result.find('div', class_='gs_a').text
|
|
85
|
+
snippet = result.find('div', class_='gs_rs').text
|
|
86
|
+
papers.append({
|
|
87
|
+
'title': title,
|
|
88
|
+
'authors': authors,
|
|
89
|
+
'snippet': snippet
|
|
90
|
+
})
|
|
91
|
+
|
|
92
|
+
return papers
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: citation-map
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.8
|
|
4
4
|
Summary: A simple tool to generate your Google Scholar citation world map.
|
|
5
5
|
Home-page: https://github.com/ChenLiu-1996/CitationMap
|
|
6
6
|
Author: Chen Liu
|
|
@@ -75,14 +75,9 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
75
75
|
output_path: str
|
|
76
76
|
(default is 'citation_map.html')
|
|
77
77
|
The path to the output HTML file.
|
|
78
|
-
|
|
78
|
+
num_processes: int
|
|
79
79
|
(default is 16)
|
|
80
80
|
Number of processes for parallel processing.
|
|
81
|
-
Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
|
|
82
|
-
num_processes_minor: int
|
|
83
|
-
(default is 32)
|
|
84
|
-
Number of processes for parallel processing.
|
|
85
|
-
Related to all other processes. Usually not causing problems.
|
|
86
81
|
use_proxy: bool
|
|
87
82
|
(default is False)
|
|
88
83
|
If true, we will use a scholarly proxy.
|
|
@@ -97,18 +92,27 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
97
92
|
```
|
|
98
93
|
|
|
99
94
|
## Debug
|
|
100
|
-
1. `scholarly._proxy_generator.MaxTriesExceededException`
|
|
95
|
+
1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
|
|
101
96
|
|
|
102
|
-
- From my experience,
|
|
97
|
+
- From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
|
|
103
98
|
- One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
|
|
104
|
-
- In case this does not help, you can try to reduce the number of processes (e.g., setting `
|
|
99
|
+
- In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
|
|
105
100
|
|
|
106
101
|
## Changelog
|
|
107
102
|
|
|
103
|
+
### Version 3.7
|
|
104
|
+
I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
|
|
105
|
+
|
|
106
|
+
**Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
|
|
107
|
+
|
|
108
108
|
### Version 3.0
|
|
109
|
-
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
109
|
+
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
110
|
+
|
|
111
|
+
For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
|
|
112
|
+
|
|
113
|
+
I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
|
|
110
114
|
|
|
111
|
-
|
|
115
|
+
**Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
|
|
112
116
|
|
|
113
117
|
### Version 2.0
|
|
114
118
|
I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
|
|
@@ -5,7 +5,7 @@ with open('README.md') as f:
|
|
|
5
5
|
|
|
6
6
|
setup(
|
|
7
7
|
name='citation-map',
|
|
8
|
-
version='3.
|
|
8
|
+
version='3.8',
|
|
9
9
|
license='Apache 2.0',
|
|
10
10
|
author='Chen Liu',
|
|
11
11
|
author_email='chen.liu.cl2482@yale.edu',
|
|
@@ -16,7 +16,7 @@ setup(
|
|
|
16
16
|
long_description_content_type='text/markdown',
|
|
17
17
|
url='https://github.com/ChenLiu-1996/CitationMap',
|
|
18
18
|
keywords='citation map, citation world map, google scholar',
|
|
19
|
-
install_requires=['scholarly', 'geopy', 'folium', 'tqdm'],
|
|
19
|
+
install_requires=['scholarly', 'geopy', 'folium', 'tqdm', 'requests', 'bs4'],
|
|
20
20
|
classifiers=[
|
|
21
21
|
'Development Status :: 3 - Alpha', # Chose either "3 - Alpha", "4 - Beta" or "5 - Production/Stable" as the current state of your package
|
|
22
22
|
'Intended Audience :: Developers', # Define that your audience are developers
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|