citation-map 3.6__tar.gz → 3.8__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: citation-map
3
- Version: 3.6
3
+ Version: 3.8
4
4
  Summary: A simple tool to generate your Google Scholar citation world map.
5
5
  Home-page: https://github.com/ChenLiu-1996/CitationMap
6
6
  Author: Chen Liu
@@ -75,14 +75,9 @@ If you open it on a browser, you will see your own version of the following cita
75
75
  output_path: str
76
76
  (default is 'citation_map.html')
77
77
  The path to the output HTML file.
78
- num_processes_major: int
78
+ num_processes: int
79
79
  (default is 16)
80
80
  Number of processes for parallel processing.
81
- Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
82
- num_processes_minor: int
83
- (default is 32)
84
- Number of processes for parallel processing.
85
- Related to all other processes. Usually not causing problems.
86
81
  use_proxy: bool
87
82
  (default is False)
88
83
  If true, we will use a scholarly proxy.
@@ -97,18 +92,27 @@ If you open it on a browser, you will see your own version of the following cita
97
92
  ```
98
93
 
99
94
  ## Debug
100
- 1. `scholarly._proxy_generator.MaxTriesExceededException`
95
+ 1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
101
96
 
102
- - From my experience, this is a good indicator that your IP address is blocked by Google Scholar due to excessive or frequent crawling (using the `scholarly` package).
97
+ - From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
103
98
  - One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
104
- - In case this does not help, you can try to reduce the number of processes (e.g., setting `num_processes_major=1`).
99
+ - In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
105
100
 
106
101
  ## Changelog
107
102
 
103
+ ### Version 3.7
104
+ I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
105
+
106
+ **Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
107
+
108
108
  ### Version 3.0
109
- I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map. For example, the string "Yale University" is a much better entry than "Assistant Professor at Yale University".
109
+ I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
110
+
111
+ For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
112
+
113
+ I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
110
114
 
111
- I applied a simple fix with some rule-based language processing. This helps us identify many missing citing locations.
115
+ **Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
112
116
 
113
117
  ### Version 2.0
114
118
  I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
@@ -59,14 +59,9 @@ If you open it on a browser, you will see your own version of the following cita
59
59
  output_path: str
60
60
  (default is 'citation_map.html')
61
61
  The path to the output HTML file.
62
- num_processes_major: int
62
+ num_processes: int
63
63
  (default is 16)
64
64
  Number of processes for parallel processing.
65
- Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
66
- num_processes_minor: int
67
- (default is 32)
68
- Number of processes for parallel processing.
69
- Related to all other processes. Usually not causing problems.
70
65
  use_proxy: bool
71
66
  (default is False)
72
67
  If true, we will use a scholarly proxy.
@@ -81,18 +76,27 @@ If you open it on a browser, you will see your own version of the following cita
81
76
  ```
82
77
 
83
78
  ## Debug
84
- 1. `scholarly._proxy_generator.MaxTriesExceededException`
79
+ 1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
85
80
 
86
- - From my experience, this is a good indicator that your IP address is blocked by Google Scholar due to excessive or frequent crawling (using the `scholarly` package).
81
+ - From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
87
82
  - One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
88
- - In case this does not help, you can try to reduce the number of processes (e.g., setting `num_processes_major=1`).
83
+ - In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
89
84
 
90
85
  ## Changelog
91
86
 
87
+ ### Version 3.7
88
+ I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
89
+
90
+ **Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
91
+
92
92
  ### Version 3.0
93
- I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map. For example, the string "Yale University" is a much better entry than "Assistant Professor at Yale University".
93
+ I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
94
+
95
+ For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
96
+
97
+ I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
94
98
 
95
- I applied a simple fix with some rule-based language processing. This helps us identify many missing citing locations.
99
+ **Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
96
100
 
97
101
  ### Version 2.0
98
102
  I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
@@ -0,0 +1,5 @@
1
+ from .citation_map import generate_citation_map
2
+ from .scholarly_support import get_author_ids
3
+
4
+ __all__ = ['generate_citation_map', 'get_author_ids']
5
+
@@ -11,6 +11,8 @@ import folium
11
11
  from tqdm import tqdm
12
12
  from multiprocessing import Pool
13
13
 
14
+ from .scholarly_support import get_author_ids
15
+
14
16
 
15
17
  def fetch_google_scholar_profile(scholar_id: str):
16
18
  '''
@@ -20,54 +22,50 @@ def fetch_google_scholar_profile(scholar_id: str):
20
22
  author = scholarly.fill(author, sections=['publications'])
21
23
  return author
22
24
 
23
- def find_all_citing_institutions(publications, num_processes_major: int = 16, num_processes_minor: int = 32) -> List[Tuple[str]]:
25
+ def find_all_citing_institutions(publications, num_processes: int = 16) -> List[Tuple[str]]:
24
26
  '''
25
- Step 2. Final all citing institutions (i.e., insitutions of citing authors).
27
+ Step 2. Find all citing institutions (i.e., insitutions of citing authors).
26
28
  '''
27
- if num_processes_minor > 1 and isinstance(num_processes_minor, int):
28
- # Fetch metadata for all publications, in parallel.
29
- with Pool(processes=num_processes_minor) as pool:
29
+ # Fetch metadata for all publications.
30
+ if num_processes > 1 and isinstance(num_processes, int):
31
+ with Pool(processes=num_processes) as pool:
30
32
  all_publications = list(tqdm(pool.imap(__fill_publication_metadata, publications),
31
- desc='Filling metadata for all publications...',
33
+ desc='Filling metadata for all publications',
32
34
  total=len(publications)))
33
35
  else:
34
36
  all_publications = []
35
- for pub in tqdm(publications, desc='Filling metadata for all publications...', total=len(publications)):
37
+ for pub in tqdm(publications, desc='Filling metadata for all publications', total=len(publications)):
36
38
  all_publications.append(__fill_publication_metadata(pub))
37
39
 
38
- if num_processes_major > 1 and isinstance(num_processes_major, int):
39
- # Find all citing papers from all publications, in parallel.
40
- with Pool(processes=num_processes_major) as pool:
41
- all_citing_papers_nested = list(tqdm(pool.imap(__citing_papers_from_publication, all_publications),
42
- desc='Finding all citations...',
43
- total=len(all_publications)))
44
- else:
45
- all_citing_papers_nested = []
46
- for pub in tqdm(all_publications, desc='Finding all citations...', total=len(all_publications)):
47
- all_citing_papers_nested.append(__citing_papers_from_publication_difficult(pub))
48
- all_citing_papers = list(itertools.chain(*all_citing_papers_nested))
49
-
50
- if num_processes_minor > 1 and isinstance(num_processes_minor, int):
51
- # Find all citing authors from all citing papers, in parallel.
52
- with Pool(processes=num_processes_minor) as pool:
53
- all_citing_authors_nested = list(tqdm(pool.imap(__authors_from_papers, all_citing_papers),
54
- desc='Finding all citing authors...',
55
- total=len(all_citing_papers)))
40
+ # Convert all publications to Google Scholar publication IDs.
41
+ # This is fast and no parallel processing is needed.
42
+ all_publication_ids = []
43
+ for pub in all_publications:
44
+ if 'cites_id' in pub:
45
+ for cites_id in pub['cites_id']:
46
+ all_publication_ids.append(cites_id)
47
+
48
+ # Find all citing authors from all publications.
49
+ if num_processes > 1 and isinstance(num_processes, int):
50
+ with Pool(processes=num_processes) as pool:
51
+ all_citing_authors_nested = list(tqdm(pool.imap(__citing_author_ids_from_publication, all_publication_ids),
52
+ desc='Finding all citing authors from cited publications',
53
+ total=len(all_publication_ids)))
56
54
  else:
57
55
  all_citing_authors_nested = []
58
- for paper in tqdm(all_citing_papers, desc='Finding all citing authors...', total=len(all_citing_papers)):
59
- all_citing_authors_nested.append(__authors_from_papers(paper))
56
+ for pub in tqdm(all_publication_ids, desc='Finding all citing authors from cited publications', total=len(all_publication_ids)):
57
+ all_citing_authors_nested.append(__citing_author_ids_from_publication(pub))
60
58
  all_citing_authors = list(itertools.chain(*all_citing_authors_nested))
61
59
 
62
- if num_processes_minor > 1 and isinstance(num_processes_minor, int):
63
- # Find all citing insitutions from all citing authors, in parallel.
64
- with Pool(processes=num_processes_minor) as pool:
60
+ # Find all citing insitutions from all citing authors.
61
+ if num_processes > 1 and isinstance(num_processes, int):
62
+ with Pool(processes=num_processes) as pool:
65
63
  author_institution_tuple_list = list(tqdm(pool.imap(__institutions_from_authors, all_citing_authors),
66
- desc='Finding all citing institutions...',
64
+ desc='Finding all citing institutions from citing authors',
67
65
  total=len(all_citing_authors)))
68
66
  else:
69
67
  author_institution_tuple_list = []
70
- for author in tqdm(all_citing_authors, desc='Finding all citing institutions...', total=len(all_citing_authors)):
68
+ for author in tqdm(all_citing_authors, desc='Finding all citing institutions from citing authors', total=len(all_citing_authors)):
71
69
  author_institution_tuple_list.append(__institutions_from_authors(author))
72
70
 
73
71
  return author_institution_tuple_list
@@ -75,6 +73,8 @@ def find_all_citing_institutions(publications, num_processes_major: int = 16, nu
75
73
  def clean_institution_names(author_institution_tuple_list: List[Tuple[str]]) -> List[Tuple[str]]:
76
74
  '''
77
75
  Step 3. Clean up the name of institutions from the author's self-entered Google Scholar affiliation.
76
+ NOTE: This logic is very naive. Please send an issue or pull request if you have any idea how to improve it.
77
+ Currently we will not consider any paid service or tools that pose extra burden on the users, such as GPT API.
78
78
  '''
79
79
  cleaned_author_institution_tuple_list = []
80
80
  for author_name, institution_string in author_institution_tuple_list:
@@ -83,32 +83,37 @@ def clean_institution_names(author_institution_tuple_list: List[Tuple[str]]) ->
83
83
  for substring in substring_list:
84
84
  # Use a regular expression to remove anything before 'at', or '@'.
85
85
  cleaned_institution = re.sub(r'.*?\bat\b|.*?@', '', substring, flags=re.IGNORECASE).strip()
86
- # Use a regular expression to check if the string is a common weird string.
87
- is_common_weird_string = re.search(
86
+ # Use a regular expression to filter out strings that represent
87
+ # a person's identity rather than affiliation.
88
+ is_common_identity_string = re.search(
88
89
  re.compile(
89
90
  r'\b(director|manager|chair|engineer|programmer|scientist|professor|lecturer|phd|student|ph\.d|department of)\b',
90
91
  re.IGNORECASE),
91
92
  substring)
92
- if not is_common_weird_string:
93
+ if not is_common_identity_string:
93
94
  cleaned_author_institution_tuple_list.append((author_name, cleaned_institution))
94
95
  return cleaned_author_institution_tuple_list
95
96
 
96
- def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]], num_processes_minor: int = 32) -> List[Tuple[str]]:
97
+ def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]], max_attempts: int = 3) -> List[Tuple[str]]:
97
98
  '''
98
99
  Step 4: Convert institutions in plain text to Geocode.
99
100
  '''
100
- if num_processes_minor > 1 and isinstance(num_processes_minor, int):
101
- with Pool(processes=num_processes_minor) as pool:
102
- coordinates = list(tqdm(pool.imap(__text_to_geocode, author_institution_tuple_list),
103
- desc='Finding all geographic coordinates...',
104
- total=len(author_institution_tuple_list)))
105
- else:
106
- coordinates = []
107
- for author_institution_tuple in tqdm(author_institution_tuple_list,
108
- desc='Finding all geographic coordinates...',
109
- total=len(author_institution_tuple_list)):
110
- coordinates.append(__text_to_geocode(author_institution_tuple))
111
-
101
+ coordinates = []
102
+ # NOTE: According to the Nomatim Usage Policy (https://operations.osmfoundation.org/policies/nominatim/),
103
+ # we are explicitly asked not to submit bulk requests on multiple threads.
104
+ # Therefore, we will keep it to a loop instead of multiprocessing.
105
+ geolocator = Nominatim(user_agent='citation_mapper')
106
+ for author_institution_tuple in tqdm(author_institution_tuple_list,
107
+ desc='Finding all geographic coordinates from citing institutions',
108
+ total=len(author_institution_tuple_list)):
109
+ for _ in range(max_attempts):
110
+ try:
111
+ author_name, institution_name = author_institution_tuple
112
+ geo_location = geolocator.geocode(institution_name)
113
+ if geo_location:
114
+ coordinates.append((geo_location.latitude, geo_location.longitude, author_name, institution_name))
115
+ except:
116
+ continue
112
117
  coordinates = [item for item in coordinates if item is not None] # Filter out empty coordinates.
113
118
  return coordinates
114
119
 
@@ -134,58 +139,23 @@ def __fill_publication_metadata(pub):
134
139
  time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
135
140
  return scholarly.fill(pub)
136
141
 
137
- def __citing_papers_from_publication(pub):
138
- if 'citedby_url' in pub:
139
- time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
140
- citing_paper_iterator = scholarly.citedby(pub)
141
- citing_paper_list = [item for item in citing_paper_iterator]
142
- return citing_paper_list
143
- return []
142
+ def __citing_author_ids_from_publication(cites_id: str):
143
+ citing_paper_search_url = 'https://scholar.google.com/scholar?hl=en&cites=' + cites_id
144
+ citing_author_ids = get_author_ids(citing_paper_search_url)
145
+ return citing_author_ids
144
146
 
145
- def __citing_papers_from_publication_difficult(pub):
146
- if 'citedby_url' in pub:
147
- time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
148
- citing_paper_iterator = scholarly.citedby(pub)
149
- citing_paper_list = []
150
- # Unpacking the iterator to a list invokes `scholarly`,
151
- # which can easily get us blocked if not handled carefully.
152
- for item in citing_paper_iterator:
153
- time.sleep(random.uniform(1, 2)) # Random delay to reduce risk of being blocked.
154
- citing_paper_list.append(item)
155
- return citing_paper_list
156
- return []
157
-
158
- def __authors_from_papers(citing_paper):
159
- if 'author_id' in citing_paper:
160
- author_id_list = citing_paper['author_id']
161
- author_id_list = [item for item in author_id_list if item] # Filter out empty author ids.
162
- return author_id_list
163
- return []
164
-
165
- def __institutions_from_authors(author_id):
147
+ def __institutions_from_authors(author_id: str):
166
148
  time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
167
149
  citing_author = scholarly.search_author_id(author_id)
168
- citing_author = scholarly.fill(citing_author)
150
+ # citing_author = scholarly.fill(citing_author)
169
151
  if 'affiliation' in citing_author:
170
152
  return (citing_author['name'], citing_author['affiliation'])
171
153
  return []
172
154
 
173
- def __text_to_geocode(author_institution_tuple, max_attempts: int = 2):
174
- geolocator = Nominatim(user_agent='citation_mapper')
175
- for _ in range(max_attempts):
176
- try:
177
- author_name, institution_name = author_institution_tuple
178
- geo_location = geolocator.geocode(institution_name)
179
- if geo_location:
180
- return (geo_location.latitude, geo_location.longitude, author_name, institution_name)
181
- except:
182
- pass
183
- return None
184
155
 
185
156
  def generate_citation_map(scholar_id: str,
186
157
  output_path: str = 'citation_map.html',
187
- num_processes_major: int = 16,
188
- num_processes_minor: int = 32,
158
+ num_processes: int = 16,
189
159
  use_proxy: bool = False,
190
160
  pin_colorful: bool = True,
191
161
  print_citing_institutions: bool = True):
@@ -199,14 +169,9 @@ def generate_citation_map(scholar_id: str,
199
169
  output_path: str
200
170
  (default is 'citation_map.html')
201
171
  The path to the output HTML file.
202
- num_processes_major: int
172
+ num_processes: int
203
173
  (default is 16)
204
174
  Number of processes for parallel processing.
205
- Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
206
- num_processes_minor: int
207
- (default is 32)
208
- Number of processes for parallel processing.
209
- Related to all other processes. Usually not causing problems.
210
175
  use_proxy: bool
211
176
  (default is False)
212
177
  If true, we will use a scholarly proxy.
@@ -230,24 +195,28 @@ def generate_citation_map(scholar_id: str,
230
195
  print('\nAuthor profile found, with %d publications.\n' % len(author_profile['publications']))
231
196
 
232
197
  author_institution_tuple_list = find_all_citing_institutions(author_profile['publications'],
233
- num_processes_major=num_processes_major,
234
- num_processes_minor=num_processes_minor)
198
+ num_processes=num_processes)
235
199
  print('\nA total of %d citing institutions recorded.\n' % len(author_institution_tuple_list))
236
200
 
201
+ # Take unique tuples.
202
+ author_institution_tuple_list = list(set(author_institution_tuple_list))
203
+
237
204
  if print_citing_institutions:
238
205
  print('\nList of all citing authors and institutions before cleaning:\n')
239
- for item in sorted(author_institution_tuple_list):
206
+ for item in sorted((author_institution_tuple_list)):
240
207
  print(item)
241
208
 
242
209
  cleaned_author_institution_tuple_list = clean_institution_names(author_institution_tuple_list)
243
210
 
211
+ # Take unique tuples.
212
+ cleaned_author_institution_tuple_list = list(set(cleaned_author_institution_tuple_list))
213
+
244
214
  if print_citing_institutions:
245
215
  print('\nList of all citing authors and institutions after cleaning:\n')
246
216
  for item in sorted(cleaned_author_institution_tuple_list):
247
217
  print(item)
248
218
 
249
- coordinates = institution_text_to_geocode(author_institution_tuple_list + cleaned_author_institution_tuple_list,
250
- num_processes_minor=num_processes_minor)
219
+ coordinates = institution_text_to_geocode(author_institution_tuple_list + cleaned_author_institution_tuple_list)
251
220
  print('\nConverted the institutions to %d Geocodes.' % len(coordinates))
252
221
 
253
222
  citation_map = create_map(coordinates, pin_colorful=pin_colorful)
@@ -260,4 +229,4 @@ if __name__ == '__main__':
260
229
  # Replace this with your Google Scholar ID.
261
230
  scholar_id = '3rDjnykAAAAJ'
262
231
  generate_citation_map(scholar_id, output_path='citation_map.html',
263
- num_processes_minor=16, use_proxy=False, pin_colorful=True, print_citing_institutions=True)
232
+ num_processes=16, use_proxy=False, pin_colorful=True, print_citing_institutions=True)
@@ -0,0 +1,92 @@
1
+ import requests
2
+ from bs4 import BeautifulSoup
3
+ from typing import List
4
+ import time
5
+ import random
6
+
7
+
8
+ def get_author_ids(paper_url: str) -> List[str]:
9
+ '''
10
+ Find the Google Scholar IDs of authors from a Google Scholar paper url.
11
+ '''
12
+
13
+ citing_author_ids = []
14
+
15
+ headers = requests.utils.default_headers()
16
+ headers.update({
17
+ 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
18
+ })
19
+
20
+ time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
21
+
22
+ # Search the url of all citing papers.
23
+ response = requests.get(paper_url, headers=headers)
24
+ if response.status_code != 200:
25
+ raise Exception('Failed to fetch the Google Scholar page')
26
+
27
+ # Get the HTML data.
28
+ soup = BeautifulSoup(response.text, 'html.parser')
29
+
30
+ # Check for common indicators of blocking
31
+ if 'CAPTCHA' in soup.text or 'not a robot' in soup.text:
32
+ print('[WARNING!] Blocked by CAPTCHA or robot check when searching %s.' % paper_url)
33
+ return []
34
+
35
+ if 'Access Denied' in soup.text or 'Forbidden' in soup.text:
36
+ print('[WARNING!] Access denied or forbidden when searching searching %s.' % paper_url)
37
+ return []
38
+
39
+ # Loop through the citation results
40
+ current_page_number = 1
41
+ while True:
42
+ for result in soup.find_all('div', class_='gs_ri'):
43
+ title_tag = result.find('h3', class_='gs_rt')
44
+ if title_tag:
45
+ author_links = result.find_all('a', href=True)
46
+ for link in author_links:
47
+ if 'user=' in link['href']:
48
+ author_id = link['href'].split('user=')[1].split('&')[0]
49
+ citing_author_ids.append(author_id)
50
+ else:
51
+ continue
52
+
53
+ # Find the page navigation.
54
+ navigation_buttons = soup.find_all('a', class_='gs_nma')
55
+ for navigation in navigation_buttons:
56
+ page_number_str = navigation.text
57
+ if page_number_str and page_number_str.isnumeric() and int(page_number_str) == current_page_number + 1:
58
+ # Found the correct button for next page.
59
+ current_page_number += 1
60
+ next_url = 'https://scholar.google.com' + navigation['href']
61
+ time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
62
+ response = requests.get(next_url, headers=headers)
63
+ if response.status_code != 200:
64
+ break
65
+ soup = BeautifulSoup(response.text, 'html.parser')
66
+ else:
67
+ continue
68
+ else:
69
+ break
70
+
71
+ return citing_author_ids
72
+
73
+ def scrape_citations(citation_url, headers):
74
+ response = requests.get(citation_url, headers=headers)
75
+ if response.status_code != 200:
76
+ raise Exception('Failed to fetch the citations page')
77
+
78
+ soup = BeautifulSoup(response.text, 'html.parser')
79
+
80
+ papers = []
81
+
82
+ for result in soup.find_all('div', class_='gs_ri'):
83
+ title = result.find('h3', class_='gs_rt').text
84
+ authors = result.find('div', class_='gs_a').text
85
+ snippet = result.find('div', class_='gs_rs').text
86
+ papers.append({
87
+ 'title': title,
88
+ 'authors': authors,
89
+ 'snippet': snippet
90
+ })
91
+
92
+ return papers
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: citation-map
3
- Version: 3.6
3
+ Version: 3.8
4
4
  Summary: A simple tool to generate your Google Scholar citation world map.
5
5
  Home-page: https://github.com/ChenLiu-1996/CitationMap
6
6
  Author: Chen Liu
@@ -75,14 +75,9 @@ If you open it on a browser, you will see your own version of the following cita
75
75
  output_path: str
76
76
  (default is 'citation_map.html')
77
77
  The path to the output HTML file.
78
- num_processes_major: int
78
+ num_processes: int
79
79
  (default is 16)
80
80
  Number of processes for parallel processing.
81
- Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
82
- num_processes_minor: int
83
- (default is 32)
84
- Number of processes for parallel processing.
85
- Related to all other processes. Usually not causing problems.
86
81
  use_proxy: bool
87
82
  (default is False)
88
83
  If true, we will use a scholarly proxy.
@@ -97,18 +92,27 @@ If you open it on a browser, you will see your own version of the following cita
97
92
  ```
98
93
 
99
94
  ## Debug
100
- 1. `scholarly._proxy_generator.MaxTriesExceededException`
95
+ 1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
101
96
 
102
- - From my experience, this is a good indicator that your IP address is blocked by Google Scholar due to excessive or frequent crawling (using the `scholarly` package).
97
+ - From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
103
98
  - One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
104
- - In case this does not help, you can try to reduce the number of processes (e.g., setting `num_processes_major=1`).
99
+ - In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
105
100
 
106
101
  ## Changelog
107
102
 
103
+ ### Version 3.7
104
+ I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
105
+
106
+ **Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
107
+
108
108
  ### Version 3.0
109
- I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map. For example, the string "Yale University" is a much better entry than "Assistant Professor at Yale University".
109
+ I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
110
+
111
+ For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
112
+
113
+ I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
110
114
 
111
- I applied a simple fix with some rule-based language processing. This helps us identify many missing citing locations.
115
+ **Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
112
116
 
113
117
  ### Version 2.0
114
118
  I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
@@ -3,6 +3,7 @@ README.md
3
3
  setup.py
4
4
  citation_map/__init__.py
5
5
  citation_map/citation_map.py
6
+ citation_map/scholarly_support.py
6
7
  citation_map.egg-info/PKG-INFO
7
8
  citation_map.egg-info/SOURCES.txt
8
9
  citation_map.egg-info/dependency_links.txt
@@ -2,3 +2,5 @@ scholarly
2
2
  geopy
3
3
  folium
4
4
  tqdm
5
+ requests
6
+ bs4
@@ -5,7 +5,7 @@ with open('README.md') as f:
5
5
 
6
6
  setup(
7
7
  name='citation-map',
8
- version='3.6',
8
+ version='3.8',
9
9
  license='Apache 2.0',
10
10
  author='Chen Liu',
11
11
  author_email='chen.liu.cl2482@yale.edu',
@@ -16,7 +16,7 @@ setup(
16
16
  long_description_content_type='text/markdown',
17
17
  url='https://github.com/ChenLiu-1996/CitationMap',
18
18
  keywords='citation map, citation world map, google scholar',
19
- install_requires=['scholarly', 'geopy', 'folium', 'tqdm'],
19
+ install_requires=['scholarly', 'geopy', 'folium', 'tqdm', 'requests', 'bs4'],
20
20
  classifiers=[
21
21
  'Development Status :: 3 - Alpha', # Chose either "3 - Alpha", "4 - Beta" or "5 - Production/Stable" as the current state of your package
22
22
  'Intended Audience :: Developers', # Define that your audience are developers
@@ -1,4 +0,0 @@
1
- from .citation_map import generate_citation_map
2
-
3
- __all__ = ['generate_citation_map']
4
-
File without changes
File without changes