citation-map 3.6__tar.gz → 3.7__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: citation-map
3
- Version: 3.6
3
+ Version: 3.7
4
4
  Summary: A simple tool to generate your Google Scholar citation world map.
5
5
  Home-page: https://github.com/ChenLiu-1996/CitationMap
6
6
  Author: Chen Liu
@@ -75,14 +75,9 @@ If you open it on a browser, you will see your own version of the following cita
75
75
  output_path: str
76
76
  (default is 'citation_map.html')
77
77
  The path to the output HTML file.
78
- num_processes_major: int
78
+ num_processes: int
79
79
  (default is 16)
80
80
  Number of processes for parallel processing.
81
- Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
82
- num_processes_minor: int
83
- (default is 32)
84
- Number of processes for parallel processing.
85
- Related to all other processes. Usually not causing problems.
86
81
  use_proxy: bool
87
82
  (default is False)
88
83
  If true, we will use a scholarly proxy.
@@ -97,18 +92,27 @@ If you open it on a browser, you will see your own version of the following cita
97
92
  ```
98
93
 
99
94
  ## Debug
100
- 1. `scholarly._proxy_generator.MaxTriesExceededException`
95
+ 1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
101
96
 
102
- - From my experience, this is a good indicator that your IP address is blocked by Google Scholar due to excessive or frequent crawling (using the `scholarly` package).
97
+ - From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
103
98
  - One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
104
- - In case this does not help, you can try to reduce the number of processes (e.g., setting `num_processes_major=1`).
99
+ - In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
105
100
 
106
101
  ## Changelog
107
102
 
103
+ ### Version 3.7
104
+ I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
105
+
106
+ **Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
107
+
108
108
  ### Version 3.0
109
- I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map. For example, the string "Yale University" is a much better entry than "Assistant Professor at Yale University".
109
+ I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
110
+
111
+ For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
112
+
113
+ I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
110
114
 
111
- I applied a simple fix with some rule-based language processing. This helps us identify many missing citing locations.
115
+ **Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
112
116
 
113
117
  ### Version 2.0
114
118
  I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
@@ -59,14 +59,9 @@ If you open it on a browser, you will see your own version of the following cita
59
59
  output_path: str
60
60
  (default is 'citation_map.html')
61
61
  The path to the output HTML file.
62
- num_processes_major: int
62
+ num_processes: int
63
63
  (default is 16)
64
64
  Number of processes for parallel processing.
65
- Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
66
- num_processes_minor: int
67
- (default is 32)
68
- Number of processes for parallel processing.
69
- Related to all other processes. Usually not causing problems.
70
65
  use_proxy: bool
71
66
  (default is False)
72
67
  If true, we will use a scholarly proxy.
@@ -81,18 +76,27 @@ If you open it on a browser, you will see your own version of the following cita
81
76
  ```
82
77
 
83
78
  ## Debug
84
- 1. `scholarly._proxy_generator.MaxTriesExceededException`
79
+ 1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
85
80
 
86
- - From my experience, this is a good indicator that your IP address is blocked by Google Scholar due to excessive or frequent crawling (using the `scholarly` package).
81
+ - From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
87
82
  - One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
88
- - In case this does not help, you can try to reduce the number of processes (e.g., setting `num_processes_major=1`).
83
+ - In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
89
84
 
90
85
  ## Changelog
91
86
 
87
+ ### Version 3.7
88
+ I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
89
+
90
+ **Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
91
+
92
92
  ### Version 3.0
93
- I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map. For example, the string "Yale University" is a much better entry than "Assistant Professor at Yale University".
93
+ I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
94
+
95
+ For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
96
+
97
+ I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
94
98
 
95
- I applied a simple fix with some rule-based language processing. This helps us identify many missing citing locations.
99
+ **Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
96
100
 
97
101
  ### Version 2.0
98
102
  I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
@@ -0,0 +1,5 @@
1
+ from .citation_map import generate_citation_map
2
+ from .scholarly_support import get_author_ids
3
+
4
+ __all__ = ['generate_citation_map', 'get_author_ids']
5
+
@@ -11,6 +11,8 @@ import folium
11
11
  from tqdm import tqdm
12
12
  from multiprocessing import Pool
13
13
 
14
+ from .scholarly_support import get_author_ids
15
+
14
16
 
15
17
  def fetch_google_scholar_profile(scholar_id: str):
16
18
  '''
@@ -20,13 +22,13 @@ def fetch_google_scholar_profile(scholar_id: str):
20
22
  author = scholarly.fill(author, sections=['publications'])
21
23
  return author
22
24
 
23
- def find_all_citing_institutions(publications, num_processes_major: int = 16, num_processes_minor: int = 32) -> List[Tuple[str]]:
25
+ def find_all_citing_institutions(publications, num_processes: int = 16) -> List[Tuple[str]]:
24
26
  '''
25
27
  Step 2. Final all citing institutions (i.e., insitutions of citing authors).
26
28
  '''
27
- if num_processes_minor > 1 and isinstance(num_processes_minor, int):
29
+ if num_processes > 1 and isinstance(num_processes, int):
28
30
  # Fetch metadata for all publications, in parallel.
29
- with Pool(processes=num_processes_minor) as pool:
31
+ with Pool(processes=num_processes) as pool:
30
32
  all_publications = list(tqdm(pool.imap(__fill_publication_metadata, publications),
31
33
  desc='Filling metadata for all publications...',
32
34
  total=len(publications)))
@@ -35,33 +37,27 @@ def find_all_citing_institutions(publications, num_processes_major: int = 16, nu
35
37
  for pub in tqdm(publications, desc='Filling metadata for all publications...', total=len(publications)):
36
38
  all_publications.append(__fill_publication_metadata(pub))
37
39
 
38
- if num_processes_major > 1 and isinstance(num_processes_major, int):
39
- # Find all citing papers from all publications, in parallel.
40
- with Pool(processes=num_processes_major) as pool:
41
- all_citing_papers_nested = list(tqdm(pool.imap(__citing_papers_from_publication, all_publications),
42
- desc='Finding all citations...',
43
- total=len(all_publications)))
44
- else:
45
- all_citing_papers_nested = []
46
- for pub in tqdm(all_publications, desc='Finding all citations...', total=len(all_publications)):
47
- all_citing_papers_nested.append(__citing_papers_from_publication_difficult(pub))
48
- all_citing_papers = list(itertools.chain(*all_citing_papers_nested))
40
+ all_publication_ids = []
41
+ for pub in all_publications:
42
+ if 'cites_id' in pub:
43
+ for cites_id in pub['cites_id']:
44
+ all_publication_ids.append(cites_id)
49
45
 
50
- if num_processes_minor > 1 and isinstance(num_processes_minor, int):
51
- # Find all citing authors from all citing papers, in parallel.
52
- with Pool(processes=num_processes_minor) as pool:
53
- all_citing_authors_nested = list(tqdm(pool.imap(__authors_from_papers, all_citing_papers),
54
- desc='Finding all citing authors...',
55
- total=len(all_citing_papers)))
46
+ if num_processes > 1 and isinstance(num_processes, int):
47
+ # Find all citing papers from all publications, in parallel.
48
+ with Pool(processes=num_processes) as pool:
49
+ all_citing_authors_nested = list(tqdm(pool.imap(__citing_author_ids_from_publication, all_publication_ids),
50
+ desc='Finding all citing authors..',
51
+ total=len(all_publication_ids)))
56
52
  else:
57
53
  all_citing_authors_nested = []
58
- for paper in tqdm(all_citing_papers, desc='Finding all citing authors...', total=len(all_citing_papers)):
59
- all_citing_authors_nested.append(__authors_from_papers(paper))
54
+ for pub in tqdm(all_publication_ids, desc='Finding all citing authors...', total=len(all_publication_ids)):
55
+ all_citing_authors_nested.append(__citing_author_ids_from_publication(pub))
60
56
  all_citing_authors = list(itertools.chain(*all_citing_authors_nested))
61
57
 
62
- if num_processes_minor > 1 and isinstance(num_processes_minor, int):
58
+ if num_processes > 1 and isinstance(num_processes, int):
63
59
  # Find all citing insitutions from all citing authors, in parallel.
64
- with Pool(processes=num_processes_minor) as pool:
60
+ with Pool(processes=num_processes) as pool:
65
61
  author_institution_tuple_list = list(tqdm(pool.imap(__institutions_from_authors, all_citing_authors),
66
62
  desc='Finding all citing institutions...',
67
63
  total=len(all_citing_authors)))
@@ -83,22 +79,23 @@ def clean_institution_names(author_institution_tuple_list: List[Tuple[str]]) ->
83
79
  for substring in substring_list:
84
80
  # Use a regular expression to remove anything before 'at', or '@'.
85
81
  cleaned_institution = re.sub(r'.*?\bat\b|.*?@', '', substring, flags=re.IGNORECASE).strip()
86
- # Use a regular expression to check if the string is a common weird string.
87
- is_common_weird_string = re.search(
82
+ # Use a regular expression to filter out strings that represent
83
+ # a person's identity rather than affiliation.
84
+ is_common_identity_string = re.search(
88
85
  re.compile(
89
86
  r'\b(director|manager|chair|engineer|programmer|scientist|professor|lecturer|phd|student|ph\.d|department of)\b',
90
87
  re.IGNORECASE),
91
88
  substring)
92
- if not is_common_weird_string:
89
+ if not is_common_identity_string:
93
90
  cleaned_author_institution_tuple_list.append((author_name, cleaned_institution))
94
91
  return cleaned_author_institution_tuple_list
95
92
 
96
- def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]], num_processes_minor: int = 32) -> List[Tuple[str]]:
93
+ def institution_text_to_geocode(author_institution_tuple_list: List[Tuple[str]], num_processes: int = 32) -> List[Tuple[str]]:
97
94
  '''
98
95
  Step 4: Convert institutions in plain text to Geocode.
99
96
  '''
100
- if num_processes_minor > 1 and isinstance(num_processes_minor, int):
101
- with Pool(processes=num_processes_minor) as pool:
97
+ if num_processes > 1 and isinstance(num_processes, int):
98
+ with Pool(processes=num_processes) as pool:
102
99
  coordinates = list(tqdm(pool.imap(__text_to_geocode, author_institution_tuple_list),
103
100
  desc='Finding all geographic coordinates...',
104
101
  total=len(author_institution_tuple_list)))
@@ -134,35 +131,12 @@ def __fill_publication_metadata(pub):
134
131
  time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
135
132
  return scholarly.fill(pub)
136
133
 
137
- def __citing_papers_from_publication(pub):
138
- if 'citedby_url' in pub:
139
- time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
140
- citing_paper_iterator = scholarly.citedby(pub)
141
- citing_paper_list = [item for item in citing_paper_iterator]
142
- return citing_paper_list
143
- return []
144
-
145
- def __citing_papers_from_publication_difficult(pub):
146
- if 'citedby_url' in pub:
147
- time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
148
- citing_paper_iterator = scholarly.citedby(pub)
149
- citing_paper_list = []
150
- # Unpacking the iterator to a list invokes `scholarly`,
151
- # which can easily get us blocked if not handled carefully.
152
- for item in citing_paper_iterator:
153
- time.sleep(random.uniform(1, 2)) # Random delay to reduce risk of being blocked.
154
- citing_paper_list.append(item)
155
- return citing_paper_list
156
- return []
134
+ def __citing_author_ids_from_publication(cites_id: str):
135
+ citing_paper_search_url = 'https://scholar.google.com/scholar?hl=en&cites=' + cites_id
136
+ citing_author_ids = get_author_ids(citing_paper_search_url)
137
+ return citing_author_ids
157
138
 
158
- def __authors_from_papers(citing_paper):
159
- if 'author_id' in citing_paper:
160
- author_id_list = citing_paper['author_id']
161
- author_id_list = [item for item in author_id_list if item] # Filter out empty author ids.
162
- return author_id_list
163
- return []
164
-
165
- def __institutions_from_authors(author_id):
139
+ def __institutions_from_authors(author_id: str):
166
140
  time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
167
141
  citing_author = scholarly.search_author_id(author_id)
168
142
  citing_author = scholarly.fill(citing_author)
@@ -170,7 +144,7 @@ def __institutions_from_authors(author_id):
170
144
  return (citing_author['name'], citing_author['affiliation'])
171
145
  return []
172
146
 
173
- def __text_to_geocode(author_institution_tuple, max_attempts: int = 2):
147
+ def __text_to_geocode(author_institution_tuple, max_attempts: int = 3):
174
148
  geolocator = Nominatim(user_agent='citation_mapper')
175
149
  for _ in range(max_attempts):
176
150
  try:
@@ -184,8 +158,7 @@ def __text_to_geocode(author_institution_tuple, max_attempts: int = 2):
184
158
 
185
159
  def generate_citation_map(scholar_id: str,
186
160
  output_path: str = 'citation_map.html',
187
- num_processes_major: int = 16,
188
- num_processes_minor: int = 32,
161
+ num_processes: int = 16,
189
162
  use_proxy: bool = False,
190
163
  pin_colorful: bool = True,
191
164
  print_citing_institutions: bool = True):
@@ -199,14 +172,9 @@ def generate_citation_map(scholar_id: str,
199
172
  output_path: str
200
173
  (default is 'citation_map.html')
201
174
  The path to the output HTML file.
202
- num_processes_major: int
175
+ num_processes: int
203
176
  (default is 16)
204
177
  Number of processes for parallel processing.
205
- Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
206
- num_processes_minor: int
207
- (default is 32)
208
- Number of processes for parallel processing.
209
- Related to all other processes. Usually not causing problems.
210
178
  use_proxy: bool
211
179
  (default is False)
212
180
  If true, we will use a scholarly proxy.
@@ -230,8 +198,7 @@ def generate_citation_map(scholar_id: str,
230
198
  print('\nAuthor profile found, with %d publications.\n' % len(author_profile['publications']))
231
199
 
232
200
  author_institution_tuple_list = find_all_citing_institutions(author_profile['publications'],
233
- num_processes_major=num_processes_major,
234
- num_processes_minor=num_processes_minor)
201
+ num_processes=num_processes)
235
202
  print('\nA total of %d citing institutions recorded.\n' % len(author_institution_tuple_list))
236
203
 
237
204
  if print_citing_institutions:
@@ -247,7 +214,7 @@ def generate_citation_map(scholar_id: str,
247
214
  print(item)
248
215
 
249
216
  coordinates = institution_text_to_geocode(author_institution_tuple_list + cleaned_author_institution_tuple_list,
250
- num_processes_minor=num_processes_minor)
217
+ num_processes=num_processes)
251
218
  print('\nConverted the institutions to %d Geocodes.' % len(coordinates))
252
219
 
253
220
  citation_map = create_map(coordinates, pin_colorful=pin_colorful)
@@ -260,4 +227,4 @@ if __name__ == '__main__':
260
227
  # Replace this with your Google Scholar ID.
261
228
  scholar_id = '3rDjnykAAAAJ'
262
229
  generate_citation_map(scholar_id, output_path='citation_map.html',
263
- num_processes_minor=16, use_proxy=False, pin_colorful=True, print_citing_institutions=True)
230
+ num_processes=16, use_proxy=False, pin_colorful=True, print_citing_institutions=True)
@@ -0,0 +1,92 @@
1
+ import requests
2
+ from bs4 import BeautifulSoup
3
+ from typing import List
4
+ import time
5
+ import random
6
+
7
+
8
+ def get_author_ids(paper_url: str) -> List[str]:
9
+ '''
10
+ Find the Google Scholar IDs of authors from a Google Scholar paper url.
11
+ '''
12
+
13
+ citing_author_ids = []
14
+
15
+ headers = requests.utils.default_headers()
16
+ headers.update({
17
+ 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
18
+ })
19
+
20
+ time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
21
+
22
+ # Search the url of all citing papers.
23
+ response = requests.get(paper_url, headers=headers)
24
+ if response.status_code != 200:
25
+ raise Exception('Failed to fetch the Google Scholar page')
26
+
27
+ # Get the HTML data.
28
+ soup = BeautifulSoup(response.text, 'html.parser')
29
+
30
+ # Check for common indicators of blocking
31
+ if 'CAPTCHA' in soup.text or 'not a robot' in soup.text:
32
+ print('[WARNING!] Blocked by CAPTCHA or robot check when searching %s.' % paper_url)
33
+ return []
34
+
35
+ if 'Access Denied' in soup.text or 'Forbidden' in soup.text:
36
+ print('[WARNING!] Access denied or forbidden when searching searching %s.' % paper_url)
37
+ return []
38
+
39
+ # Loop through the citation results
40
+ current_page_number = 1
41
+ while True:
42
+ for result in soup.find_all('div', class_='gs_ri'):
43
+ title_tag = result.find('h3', class_='gs_rt')
44
+ if title_tag:
45
+ author_links = result.find_all('a', href=True)
46
+ for link in author_links:
47
+ if 'user=' in link['href']:
48
+ author_id = link['href'].split('user=')[1].split('&')[0]
49
+ citing_author_ids.append(author_id)
50
+ else:
51
+ continue
52
+
53
+ # Find the page navigation.
54
+ navigation_buttons = soup.find_all('a', class_='gs_nma')
55
+ for navigation in navigation_buttons:
56
+ page_number_str = navigation.text
57
+ if page_number_str and page_number_str.isnumeric() and int(page_number_str) == current_page_number + 1:
58
+ # Found the correct button for next page.
59
+ current_page_number += 1
60
+ next_url = 'https://scholar.google.com' + navigation['href']
61
+ time.sleep(random.uniform(1, 5)) # Random delay to reduce risk of being blocked.
62
+ response = requests.get(next_url, headers=headers)
63
+ if response.status_code != 200:
64
+ break
65
+ soup = BeautifulSoup(response.text, 'html.parser')
66
+ else:
67
+ continue
68
+ else:
69
+ break
70
+
71
+ return citing_author_ids
72
+
73
+ def scrape_citations(citation_url, headers):
74
+ response = requests.get(citation_url, headers=headers)
75
+ if response.status_code != 200:
76
+ raise Exception('Failed to fetch the citations page')
77
+
78
+ soup = BeautifulSoup(response.text, 'html.parser')
79
+
80
+ papers = []
81
+
82
+ for result in soup.find_all('div', class_='gs_ri'):
83
+ title = result.find('h3', class_='gs_rt').text
84
+ authors = result.find('div', class_='gs_a').text
85
+ snippet = result.find('div', class_='gs_rs').text
86
+ papers.append({
87
+ 'title': title,
88
+ 'authors': authors,
89
+ 'snippet': snippet
90
+ })
91
+
92
+ return papers
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: citation-map
3
- Version: 3.6
3
+ Version: 3.7
4
4
  Summary: A simple tool to generate your Google Scholar citation world map.
5
5
  Home-page: https://github.com/ChenLiu-1996/CitationMap
6
6
  Author: Chen Liu
@@ -75,14 +75,9 @@ If you open it on a browser, you will see your own version of the following cita
75
75
  output_path: str
76
76
  (default is 'citation_map.html')
77
77
  The path to the output HTML file.
78
- num_processes_major: int
78
+ num_processes: int
79
79
  (default is 16)
80
80
  Number of processes for parallel processing.
81
- Related to `scholarly.citedby()`. Primary reason for Google Scholar bans.
82
- num_processes_minor: int
83
- (default is 32)
84
- Number of processes for parallel processing.
85
- Related to all other processes. Usually not causing problems.
86
81
  use_proxy: bool
87
82
  (default is False)
88
83
  If true, we will use a scholarly proxy.
@@ -97,18 +92,27 @@ If you open it on a browser, you will see your own version of the following cita
97
92
  ```
98
93
 
99
94
  ## Debug
100
- 1. `scholarly._proxy_generator.MaxTriesExceededException`
95
+ 1. `scholarly._proxy_generator.MaxTriesExceededException` or `[WARNING!] Blocked by CAPTCHA or robot check`
101
96
 
102
- - From my experience, this is a good indicator that your IP address is blocked by Google Scholar due to excessive or frequent crawling (using the `scholarly` package).
97
+ - From my experience, both are good indicators that your IP address is blocked by Google Scholar due to excessive crawling (using the `scholarly` package).
103
98
  - One hot fix I found was to hop on a University VPN and run again. I typically experience this error after running the tool twice, and I need to disconnect and reconnect my VPN to "unblock" myself.
104
- - In case this does not help, you can try to reduce the number of processes (e.g., setting `num_processes_major=1`).
99
+ - In case this does not help, you can try to change IP adress and reduce the number of processes (e.g., setting `num_processes=1`).
105
100
 
106
101
  ## Changelog
107
102
 
103
+ ### Version 3.7
104
+ I updated the logic for webscraping and avoided using `scholarly.citeby()` which is the biggest trigger of blacklisting from Google Scholar.
105
+
106
+ **Now we should be able to handle users with more citations than before. I tested on a profile with 400 citations without running into any issue.**
107
+
108
108
  ### Version 3.0
109
- I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations are not successfully found in the system and hence are not converted to geographic coordinates on the world map. For example, the string "Yale University" is a much better entry than "Assistant Professor at Yale University".
109
+ I realized a problem with how I used `geopy.geocoders`. A majority of the authors' self-entered affiliations include details that are irrelevant to the affiliation itself. Therefore, they are not successfully found in the system and hence are not converted to geographic coordinates on the world map.
110
+
111
+ For example, we would want the substring "Yale University" from the string "Assistant Professor at Yale University".
112
+
113
+ I applied a simple fix with some rule-based natural language processing. This helps us identify many missing citing locations.
110
114
 
111
- I applied a simple fix with some rule-based language processing. This helps us identify many missing citing locations.
115
+ **Please raise an issue or submit a pull request if you have some good idea to better process the affiliation string. Note that currently I am not considering any paid service or tools that pose extra burden on the users, such as GPT API.**
112
116
 
113
117
  ### Version 2.0
114
118
  I finally managed to **drastically speed up** the process using multiprocessing, in a way that avoids being blocked by Google Scholar.
@@ -3,6 +3,7 @@ README.md
3
3
  setup.py
4
4
  citation_map/__init__.py
5
5
  citation_map/citation_map.py
6
+ citation_map/scholarly_support.py
6
7
  citation_map.egg-info/PKG-INFO
7
8
  citation_map.egg-info/SOURCES.txt
8
9
  citation_map.egg-info/dependency_links.txt
@@ -2,3 +2,5 @@ scholarly
2
2
  geopy
3
3
  folium
4
4
  tqdm
5
+ requests
6
+ bs4
@@ -5,7 +5,7 @@ with open('README.md') as f:
5
5
 
6
6
  setup(
7
7
  name='citation-map',
8
- version='3.6',
8
+ version='3.7',
9
9
  license='Apache 2.0',
10
10
  author='Chen Liu',
11
11
  author_email='chen.liu.cl2482@yale.edu',
@@ -16,7 +16,7 @@ setup(
16
16
  long_description_content_type='text/markdown',
17
17
  url='https://github.com/ChenLiu-1996/CitationMap',
18
18
  keywords='citation map, citation world map, google scholar',
19
- install_requires=['scholarly', 'geopy', 'folium', 'tqdm'],
19
+ install_requires=['scholarly', 'geopy', 'folium', 'tqdm', 'requests', 'bs4'],
20
20
  classifiers=[
21
21
  'Development Status :: 3 - Alpha', # Chose either "3 - Alpha", "4 - Beta" or "5 - Production/Stable" as the current state of your package
22
22
  'Intended Audience :: Developers', # Define that your audience are developers
@@ -1,4 +0,0 @@
1
- from .citation_map import generate_citation_map
2
-
3
- __all__ = ['generate_citation_map']
4
-
File without changes
File without changes