citation-map 3.6__tar.gz → 3.7__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {citation-map-3.6 → citation-map-3.7}/PKG-INFO +16 -12
- {citation-map-3.6 → citation-map-3.7}/README.md +15 -11
- citation-map-3.7/citation_map/__init__.py +5 -0
- {citation-map-3.6 → citation-map-3.7}/citation_map/citation_map.py +38 -71
- citation-map-3.7/citation_map/scholarly_support.py +92 -0
- {citation-map-3.6 → citation-map-3.7}/citation_map.egg-info/PKG-INFO +16 -12
- {citation-map-3.6 → citation-map-3.7}/citation_map.egg-info/SOURCES.txt +1 -0
- {citation-map-3.6 → citation-map-3.7}/citation_map.egg-info/requires.txt +2 -0
- {citation-map-3.6 → citation-map-3.7}/setup.py +2 -2
- citation-map-3.6/citation_map/__init__.py +0 -4
- {citation-map-3.6 → citation-map-3.7}/LICENSE.md +0 -0
- {citation-map-3.6 → citation-map-3.7}/citation_map.egg-info/dependency_links.txt +0 -0
- {citation-map-3.6 → citation-map-3.7}/citation_map.egg-info/top_level.txt +0 -0
- {citation-map-3.6 → citation-map-3.7}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: citation-map
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.7
|
|
4
4
|
Summary: A simple tool to generate your Google Scholar citation world map.
|
|
5
5
|
Home-page: https://github.com/ChenLiu-1996/CitationMap
|
|
6
6
|
Author: Chen Liu
|
|
@@ -75,14 +75,9 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
75
75
|
output_path: str
|
|
76
76
|
(default is 'citation_map.html')
|
|
77
77
|
The path to the output HTML file.
|
|
78
|
-
|
|
78
|
+
num_processes: int
|
|
79
79
|
(default is 16)
|
|
80
80
|
Number of processes for parallel processing.
|
|
81
|
-
Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
|
|
82
|
-
num_processes_minor: int
|
|
83
|
-
(default is 32)
|
|
84
|
-
Number of processes for parallel processing.
|
|
85
|
-
Related to all other processes. Usually not causing problems.
|
|
86
81
|
use_proxy: bool
|
|
87
82
|
(default is False)
|
|
88
83
|
If true, we will use a scholarly proxy.
|
|
@@ -97,18 +92,27 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
97
92
|
```
|
|
98
93
|
|
|
99
94
|
## Debug
|
|
100
|
-
1. `scholarly._proxy_generator.MaxTriesExceededException`
|
|
95
|
+
1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
|
|
101
96
|
|
|
102
|
-
- From my experience,
|
|
97
|
+
- From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
|
|
103
98
|
- One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
|
|
104
|
-
- In case this does not help, you can try to reduce the number of processes (e.g., setting `
|
|
99
|
+
- In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
|
|
105
100
|
|
|
106
101
|
## Changelog
|
|
107
102
|
|
|
103
|
+
### Version 3.7
|
|
104
|
+
I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
|
|
105
|
+
|
|
106
|
+
**Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
|
|
107
|
+
|
|
108
108
|
### Version 3.0
|
|
109
|
-
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
109
|
+
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
110
|
+
|
|
111
|
+
For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
|
|
112
|
+
|
|
113
|
+
I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
|
|
110
114
|
|
|
111
|
-
|
|
115
|
+
**Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
|
|
112
116
|
|
|
113
117
|
### Version 2.0
|
|
114
118
|
I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
|
|
@@ -59,14 +59,9 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
59
59
|
output_path: str
|
|
60
60
|
(default is 'citation_map.html')
|
|
61
61
|
The path to the output HTML file.
|
|
62
|
-
|
|
62
|
+
num_processes: int
|
|
63
63
|
(default is 16)
|
|
64
64
|
Number of processes for parallel processing.
|
|
65
|
-
Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
|
|
66
|
-
num_processes_minor: int
|
|
67
|
-
(default is 32)
|
|
68
|
-
Number of processes for parallel processing.
|
|
69
|
-
Related to all other processes. Usually not causing problems.
|
|
70
65
|
use_proxy: bool
|
|
71
66
|
(default is False)
|
|
72
67
|
If true, we will use a scholarly proxy.
|
|
@@ -81,18 +76,27 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
81
76
|
```
|
|
82
77
|
|
|
83
78
|
## Debug
|
|
84
|
-
1. `scholarly._proxy_generator.MaxTriesExceededException`
|
|
79
|
+
1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
|
|
85
80
|
|
|
86
|
-
- From my experience,
|
|
81
|
+
- From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
|
|
87
82
|
- One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
|
|
88
|
-
- In case this does not help, you can try to reduce the number of processes (e.g., setting `
|
|
83
|
+
- In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
|
|
89
84
|
|
|
90
85
|
## Changelog
|
|
91
86
|
|
|
87
|
+
### Version 3.7
|
|
88
|
+
I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
|
|
89
|
+
|
|
90
|
+
**Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
|
|
91
|
+
|
|
92
92
|
### Version 3.0
|
|
93
|
-
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
93
|
+
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
94
|
+
|
|
95
|
+
For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
|
|
96
|
+
|
|
97
|
+
I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
|
|
94
98
|
|
|
95
|
-
|
|
99
|
+
**Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
|
|
96
100
|
|
|
97
101
|
### Version 2.0
|
|
98
102
|
I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
|
|
@@ -11,6 +11,8 @@ import folium
|
|
|
11
11
|
from tqdm import tqdm
|
|
12
12
|
from multiprocessing import Pool
|
|
13
13
|
|
|
14
|
+
from .scholarly_support import get_author_ids
|
|
15
|
+
|
|
14
16
|
|
|
15
17
|
def fetch_google_scholar_profile(scholar_id: str):
|
|
16
18
|
'''
|
|
@@ -20,13 +22,13 @@ def fetch_google_scholar_profile(scholar_id: str):
|
|
|
20
22
|
author = scholarly.fill(author, sections=['publications'])
|
|
21
23
|
return author
|
|
22
24
|
|
|
23
|
-
def find_all_citing_institutions(publications,
|
|
25
|
+
def find_all_citing_institutions(publications, num_processes: int = 16) -> List[Tuple[str]]:
|
|
24
26
|
'''
|
|
25
27
|
Step 2. Final all citing institutions (i.e., insitutions of citing authors).
|
|
26
28
|
'''
|
|
27
|
-
if
|
|
29
|
+
if num_processes > 1 and isinstance(num_processes, int):
|
|
28
30
|
# Fetch metadata for all publications, in parallel.
|
|
29
|
-
with Pool(processes=
|
|
31
|
+
with Pool(processes=num_processes) as pool:
|
|
30
32
|
all_publications = list(tqdm(pool.imap(__fill_publication_metadata, publications),
|
|
31
33
|
desc='Filling metadata for all publications...',
|
|
32
34
|
total=len(publications)))
|
|
@@ -35,33 +37,27 @@ def find_all_citing_institutions(publications, num_processes_major: int = 16, nu
|
|
|
35
37
|
for pub in tqdm(publications, desc='Filling metadata for all publications...', total=len(publications)):
|
|
36
38
|
all_publications.append(__fill_publication_metadata(pub))
|
|
37
39
|
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
total=len(all_publications)))
|
|
44
|
-
else:
|
|
45
|
-
all_citing_papers_nested = []
|
|
46
|
-
for pub in tqdm(all_publications, desc='Finding all citations...', total=len(all_publications)):
|
|
47
|
-
all_citing_papers_nested.append(__citing_papers_from_publication_difficult(pub))
|
|
48
|
-
all_citing_papers = list(itertools.chain(*all_citing_papers_nested))
|
|
40
|
+
all_publication_ids = []
|
|
41
|
+
for pub in all_publications:
|
|
42
|
+
if 'cites_id' in pub:
|
|
43
|
+
for cites_id in pub['cites_id']:
|
|
44
|
+
all_publication_ids.append(cites_id)
|
|
49
45
|
|
|
50
|
-
if
|
|
51
|
-
# Find all citing
|
|
52
|
-
with Pool(processes=
|
|
53
|
-
all_citing_authors_nested = list(tqdm(pool.imap(
|
|
54
|
-
|
|
55
|
-
|
|
46
|
+
if num_processes > 1 and isinstance(num_processes, int):
|
|
47
|
+
# Find all citing papers from all publications, in parallel.
|
|
48
|
+
with Pool(processes=num_processes) as pool:
|
|
49
|
+
all_citing_authors_nested = list(tqdm(pool.imap(__citing_author_ids_from_publication, all_publication_ids),
|
|
50
|
+
desc='Finding all citing authors..',
|
|
51
|
+
total=len(all_publication_ids)))
|
|
56
52
|
else:
|
|
57
53
|
all_citing_authors_nested = []
|
|
58
|
-
for
|
|
59
|
-
all_citing_authors_nested.append(
|
|
54
|
+
for pub in tqdm(all_publication_ids, desc='Finding all citing authors...', total=len(all_publication_ids)):
|
|
55
|
+
all_citing_authors_nested.append(__citing_author_ids_from_publication(pub))
|
|
60
56
|
all_citing_authors = list(itertools.chain(*all_citing_authors_nested))
|
|
61
57
|
|
|
62
|
-
if
|
|
58
|
+
if num_processes > 1 and isinstance(num_processes, int):
|
|
63
59
|
# Find all citing insitutions from all citing authors, in parallel.
|
|
64
|
-
with Pool(processes=
|
|
60
|
+
with Pool(processes=num_processes) as pool:
|
|
65
61
|
author_institution_tuple_list = list(tqdm(pool.imap(__institutions_from_authors, all_citing_authors),
|
|
66
62
|
desc='Finding all citing institutions...',
|
|
67
63
|
total=len(all_citing_authors)))
|
|
@@ -83,22 +79,23 @@ def clean_institution_names(author_institution_tuple_list: List[Tuple[str]]) ->
|
|
|
83
79
|
for substring in substring_list:
|
|
84
80
|
# Use a regular expression to remove anything before 'at', or '@'.
|
|
85
81
|
cleaned_institution = re.sub(r'.*?\bat\b|.*?@', '', substring, flags=re.IGNORECASE).strip()
|
|
86
|
-
# Use a regular expression to
|
|
87
|
-
|
|
82
|
+
# Use a regular expression to filter out strings that represent
|
|
83
|
+
# a person's identity rather than affiliation.
|
|
84
|
+
is_common_identity_string = re.search(
|
|
88
85
|
re.compile(
|
|
89
86
|
r'\b(director|manager|chair|engineer|programmer|scientist|professor|lecturer|phd|student|ph\.d|department of)\b',
|
|
90
87
|
re.IGNORECASE),
|
|
91
88
|
substring)
|
|
92
|
-
if not
|
|
89
|
+
if not is_common_identity_string:
|
|
93
90
|
cleaned_author_institution_tuple_list.append((author_name, cleaned_institution))
|
|
94
91
|
return cleaned_author_institution_tuple_list
|
|
95
92
|
|
|
96
|
-
def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]],
|
|
93
|
+
def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]], num_processes: int = 32) -> List[Tuple[str]]:
|
|
97
94
|
'''
|
|
98
95
|
Step 4: Convert institutions in plain text to Geocode.
|
|
99
96
|
'''
|
|
100
|
-
if
|
|
101
|
-
with Pool(processes=
|
|
97
|
+
if num_processes > 1 and isinstance(num_processes, int):
|
|
98
|
+
with Pool(processes=num_processes) as pool:
|
|
102
99
|
coordinates = list(tqdm(pool.imap(__text_to_geocode, author_institution_tuple_list),
|
|
103
100
|
desc='Finding all geographic coordinates...',
|
|
104
101
|
total=len(author_institution_tuple_list)))
|
|
@@ -134,35 +131,12 @@ def __fill_publication_metadata(pub):
|
|
|
134
131
|
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
135
132
|
return scholarly.fill(pub)
|
|
136
133
|
|
|
137
|
-
def
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
citing_paper_list = [item for item in citing_paper_iterator]
|
|
142
|
-
return citing_paper_list
|
|
143
|
-
return []
|
|
144
|
-
|
|
145
|
-
def __citing_papers_from_publication_difficult(pub):
|
|
146
|
-
if 'citedby_url' in pub:
|
|
147
|
-
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
148
|
-
citing_paper_iterator = scholarly.citedby(pub)
|
|
149
|
-
citing_paper_list = []
|
|
150
|
-
# Unpacking the iterator to a list invokes `scholarly`,
|
|
151
|
-
# which can easily get us blocked if not handled carefully.
|
|
152
|
-
for item in citing_paper_iterator:
|
|
153
|
-
time.sleep(random.uniform(1, 2)) # Random delay to reduce risk of being blocked.
|
|
154
|
-
citing_paper_list.append(item)
|
|
155
|
-
return citing_paper_list
|
|
156
|
-
return []
|
|
134
|
+
def __citing_author_ids_from_publication(cites_id: str):
|
|
135
|
+
citing_paper_search_url = 'https://scholar.google.com/scholar?hl=en&cites=' + cites_id
|
|
136
|
+
citing_author_ids = get_author_ids(citing_paper_search_url)
|
|
137
|
+
return citing_author_ids
|
|
157
138
|
|
|
158
|
-
def
|
|
159
|
-
if 'author_id' in citing_paper:
|
|
160
|
-
author_id_list = citing_paper['author_id']
|
|
161
|
-
author_id_list = [item for item in author_id_list if item] # Filter out empty author ids.
|
|
162
|
-
return author_id_list
|
|
163
|
-
return []
|
|
164
|
-
|
|
165
|
-
def __institutions_from_authors(author_id):
|
|
139
|
+
def __institutions_from_authors(author_id: str):
|
|
166
140
|
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
167
141
|
citing_author = scholarly.search_author_id(author_id)
|
|
168
142
|
citing_author = scholarly.fill(citing_author)
|
|
@@ -170,7 +144,7 @@ def __institutions_from_authors(author_id):
|
|
|
170
144
|
return (citing_author['name'], citing_author['affiliation'])
|
|
171
145
|
return []
|
|
172
146
|
|
|
173
|
-
def __text_to_geocode(author_institution_tuple, max_attempts: int =
|
|
147
|
+
def __text_to_geocode(author_institution_tuple, max_attempts: int = 3):
|
|
174
148
|
geolocator = Nominatim(user_agent='citation_mapper')
|
|
175
149
|
for _ in range(max_attempts):
|
|
176
150
|
try:
|
|
@@ -184,8 +158,7 @@ def __text_to_geocode(author_institution_tuple, max_attempts: int = 2):
|
|
|
184
158
|
|
|
185
159
|
def generate_citation_map(scholar_id: str,
|
|
186
160
|
output_path: str = 'citation_map.html',
|
|
187
|
-
|
|
188
|
-
num_processes_minor: int = 32,
|
|
161
|
+
num_processes: int = 16,
|
|
189
162
|
use_proxy: bool = False,
|
|
190
163
|
pin_colorful: bool = True,
|
|
191
164
|
print_citing_institutions: bool = True):
|
|
@@ -199,14 +172,9 @@ def generate_citation_map(scholar_id: str,
|
|
|
199
172
|
output_path: str
|
|
200
173
|
(default is 'citation_map.html')
|
|
201
174
|
The path to the output HTML file.
|
|
202
|
-
|
|
175
|
+
num_processes: int
|
|
203
176
|
(default is 16)
|
|
204
177
|
Number of processes for parallel processing.
|
|
205
|
-
Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
|
|
206
|
-
num_processes_minor: int
|
|
207
|
-
(default is 32)
|
|
208
|
-
Number of processes for parallel processing.
|
|
209
|
-
Related to all other processes. Usually not causing problems.
|
|
210
178
|
use_proxy: bool
|
|
211
179
|
(default is False)
|
|
212
180
|
If true, we will use a scholarly proxy.
|
|
@@ -230,8 +198,7 @@ def generate_citation_map(scholar_id: str,
|
|
|
230
198
|
print('\nAuthor profile found, with %d publications.\n' % len(author_profile['publications']))
|
|
231
199
|
|
|
232
200
|
author_institution_tuple_list = find_all_citing_institutions(author_profile['publications'],
|
|
233
|
-
|
|
234
|
-
num_processes_minor=num_processes_minor)
|
|
201
|
+
num_processes=num_processes)
|
|
235
202
|
print('\nA total of %d citing institutions recorded.\n' % len(author_institution_tuple_list))
|
|
236
203
|
|
|
237
204
|
if print_citing_institutions:
|
|
@@ -247,7 +214,7 @@ def generate_citation_map(scholar_id: str,
|
|
|
247
214
|
print(item)
|
|
248
215
|
|
|
249
216
|
coordinates = institution_text_to_geocode(author_institution_tuple_list + cleaned_author_institution_tuple_list,
|
|
250
|
-
|
|
217
|
+
num_processes=num_processes)
|
|
251
218
|
print('\nConverted the institutions to %d Geocodes.' % len(coordinates))
|
|
252
219
|
|
|
253
220
|
citation_map = create_map(coordinates, pin_colorful=pin_colorful)
|
|
@@ -260,4 +227,4 @@ if __name__ == '__main__':
|
|
|
260
227
|
# Replace this with your Google Scholar ID.
|
|
261
228
|
scholar_id = '3rDjnykAAAAJ'
|
|
262
229
|
generate_citation_map(scholar_id, output_path='citation_map.html',
|
|
263
|
-
|
|
230
|
+
num_processes=16, use_proxy=False, pin_colorful=True, print_citing_institutions=True)
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
import requests
|
|
2
|
+
from bs4 import BeautifulSoup
|
|
3
|
+
from typing import List
|
|
4
|
+
import time
|
|
5
|
+
import random
|
|
6
|
+
|
|
7
|
+
|
|
8
|
+
def get_author_ids(paper_url: str) -> List[str]:
|
|
9
|
+
'''
|
|
10
|
+
Find the Google Scholar IDs of authors from a Google Scholar paper url.
|
|
11
|
+
'''
|
|
12
|
+
|
|
13
|
+
citing_author_ids = []
|
|
14
|
+
|
|
15
|
+
headers = requests.utils.default_headers()
|
|
16
|
+
headers.update({
|
|
17
|
+
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
|
|
18
|
+
})
|
|
19
|
+
|
|
20
|
+
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
21
|
+
|
|
22
|
+
# Search the url of all citing papers.
|
|
23
|
+
response = requests.get(paper_url, headers=headers)
|
|
24
|
+
if response.status_code != 200:
|
|
25
|
+
raise Exception('Failed to fetch the Google Scholar page')
|
|
26
|
+
|
|
27
|
+
# Get the HTML data.
|
|
28
|
+
soup = BeautifulSoup(response.text, 'html.parser')
|
|
29
|
+
|
|
30
|
+
# Check for common indicators of blocking
|
|
31
|
+
if 'CAPTCHA' in soup.text or 'not a robot' in soup.text:
|
|
32
|
+
print('[WARNING!] Blocked by CAPTCHA or robot check when searching %s.' % paper_url)
|
|
33
|
+
return []
|
|
34
|
+
|
|
35
|
+
if 'Access Denied' in soup.text or 'Forbidden' in soup.text:
|
|
36
|
+
print('[WARNING!] Access denied or forbidden when searching searching %s.' % paper_url)
|
|
37
|
+
return []
|
|
38
|
+
|
|
39
|
+
# Loop through the citation results
|
|
40
|
+
current_page_number = 1
|
|
41
|
+
while True:
|
|
42
|
+
for result in soup.find_all('div', class_='gs_ri'):
|
|
43
|
+
title_tag = result.find('h3', class_='gs_rt')
|
|
44
|
+
if title_tag:
|
|
45
|
+
author_links = result.find_all('a', href=True)
|
|
46
|
+
for link in author_links:
|
|
47
|
+
if 'user=' in link['href']:
|
|
48
|
+
author_id = link['href'].split('user=')[1].split('&')[0]
|
|
49
|
+
citing_author_ids.append(author_id)
|
|
50
|
+
else:
|
|
51
|
+
continue
|
|
52
|
+
|
|
53
|
+
# Find the page navigation.
|
|
54
|
+
navigation_buttons = soup.find_all('a', class_='gs_nma')
|
|
55
|
+
for navigation in navigation_buttons:
|
|
56
|
+
page_number_str = navigation.text
|
|
57
|
+
if page_number_str and page_number_str.isnumeric() and int(page_number_str) == current_page_number + 1:
|
|
58
|
+
# Found the correct button for next page.
|
|
59
|
+
current_page_number += 1
|
|
60
|
+
next_url = 'https://scholar.google.com' + navigation['href']
|
|
61
|
+
time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
|
|
62
|
+
response = requests.get(next_url, headers=headers)
|
|
63
|
+
if response.status_code != 200:
|
|
64
|
+
break
|
|
65
|
+
soup = BeautifulSoup(response.text, 'html.parser')
|
|
66
|
+
else:
|
|
67
|
+
continue
|
|
68
|
+
else:
|
|
69
|
+
break
|
|
70
|
+
|
|
71
|
+
return citing_author_ids
|
|
72
|
+
|
|
73
|
+
def scrape_citations(citation_url, headers):
|
|
74
|
+
response = requests.get(citation_url, headers=headers)
|
|
75
|
+
if response.status_code != 200:
|
|
76
|
+
raise Exception('Failed to fetch the citations page')
|
|
77
|
+
|
|
78
|
+
soup = BeautifulSoup(response.text, 'html.parser')
|
|
79
|
+
|
|
80
|
+
papers = []
|
|
81
|
+
|
|
82
|
+
for result in soup.find_all('div', class_='gs_ri'):
|
|
83
|
+
title = result.find('h3', class_='gs_rt').text
|
|
84
|
+
authors = result.find('div', class_='gs_a').text
|
|
85
|
+
snippet = result.find('div', class_='gs_rs').text
|
|
86
|
+
papers.append({
|
|
87
|
+
'title': title,
|
|
88
|
+
'authors': authors,
|
|
89
|
+
'snippet': snippet
|
|
90
|
+
})
|
|
91
|
+
|
|
92
|
+
return papers
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: citation-map
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.7
|
|
4
4
|
Summary: A simple tool to generate your Google Scholar citation world map.
|
|
5
5
|
Home-page: https://github.com/ChenLiu-1996/CitationMap
|
|
6
6
|
Author: Chen Liu
|
|
@@ -75,14 +75,9 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
75
75
|
output_path: str
|
|
76
76
|
(default is 'citation_map.html')
|
|
77
77
|
The path to the output HTML file.
|
|
78
|
-
|
|
78
|
+
num_processes: int
|
|
79
79
|
(default is 16)
|
|
80
80
|
Number of processes for parallel processing.
|
|
81
|
-
Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
|
|
82
|
-
num_processes_minor: int
|
|
83
|
-
(default is 32)
|
|
84
|
-
Number of processes for parallel processing.
|
|
85
|
-
Related to all other processes. Usually not causing problems.
|
|
86
81
|
use_proxy: bool
|
|
87
82
|
(default is False)
|
|
88
83
|
If true, we will use a scholarly proxy.
|
|
@@ -97,18 +92,27 @@ If you open it on a browser, you will see your own version of the following cita
|
|
|
97
92
|
```
|
|
98
93
|
|
|
99
94
|
## Debug
|
|
100
|
-
1. `scholarly._proxy_generator.MaxTriesExceededException`
|
|
95
|
+
1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
|
|
101
96
|
|
|
102
|
-
- From my experience,
|
|
97
|
+
- From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
|
|
103
98
|
- One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
|
|
104
|
-
- In case this does not help, you can try to reduce the number of processes (e.g., setting `
|
|
99
|
+
- In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
|
|
105
100
|
|
|
106
101
|
## Changelog
|
|
107
102
|
|
|
103
|
+
### Version 3.7
|
|
104
|
+
I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
|
|
105
|
+
|
|
106
|
+
**Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
|
|
107
|
+
|
|
108
108
|
### Version 3.0
|
|
109
|
-
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
109
|
+
I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
|
|
110
|
+
|
|
111
|
+
For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
|
|
112
|
+
|
|
113
|
+
I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
|
|
110
114
|
|
|
111
|
-
|
|
115
|
+
**Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
|
|
112
116
|
|
|
113
117
|
### Version 2.0
|
|
114
118
|
I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
|
|
@@ -5,7 +5,7 @@ with open('README.md') as f:
|
|
|
5
5
|
|
|
6
6
|
setup(
|
|
7
7
|
name='citation-map',
|
|
8
|
-
version='3.
|
|
8
|
+
version='3.7',
|
|
9
9
|
license='Apache 2.0',
|
|
10
10
|
author='Chen Liu',
|
|
11
11
|
author_email='chen.liu.cl2482@yale.edu',
|
|
@@ -16,7 +16,7 @@ setup(
|
|
|
16
16
|
long_description_content_type='text/markdown',
|
|
17
17
|
url='https://github.com/ChenLiu-1996/CitationMap',
|
|
18
18
|
keywords='citation map, citation world map, google scholar',
|
|
19
|
-
install_requires=['scholarly', 'geopy', 'folium', 'tqdm'],
|
|
19
|
+
install_requires=['scholarly', 'geopy', 'folium', 'tqdm', 'requests', 'bs4'],
|
|
20
20
|
classifiers=[
|
|
21
21
|
'Development Status :: 3 - Alpha', # Chose either "3 - Alpha", "4 - Beta" or "5 - Production/Stable" as the current state of your package
|
|
22
22
|
'Intended Audience :: Developers', # Define that your audience are developers
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|