paperscraper 0.2.12__tar.gz → 0.2.13__tar.gz

{paperscraper-0.2.12 → paperscraper-0.2.13}/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.1
 Name: paperscraper
-Version: 0.2.12
+Version: 0.2.13
 Summary: paperscraper: Package to scrape papers.
-Home-page: https://github.com/PhosphorylatedRabbits/paperscraper
+Home-page: https://github.com/jannisborn/paperscraper
 Author: Jannis Born, Matteo Manica
 Author-email: jannis.born@gmx.de, drugilsberg@gmail.com
 License: MIT
@@ -22,7 +22,7 @@ Requires-Dist: pymed
 Requires-Dist: pandas
 Requires-Dist: requests
 Requires-Dist: tqdm
-Requires-Dist: scholarly==0.5.1
+Requires-Dist: scholarly>=1.0.0
 Requires-Dist: seaborn
 Requires-Dist: matplotlib
 Requires-Dist: matplotlib_venn
@@ -32,8 +32,8 @@ Requires-Dist: thefuzz
 Requires-Dist: pytest
 
 
-[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml/badge.svg)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml)
-[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml/badge.svg)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml)
+[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml/badge.svg?branch=main)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml?query=branch%3Amain)
+[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml/badge.svg?branch=main)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml?query=branch%3Amain)
 [![License:
 MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![PyPI version](https://badge.fury.io/py/paperscraper.svg)](https://badge.fury.io/py/paperscraper)
@@ -70,9 +70,9 @@ biorxiv()  # Takes ~1h and should result in ~350 MB file
 chemrxiv()  #  Takes ~45min and should result in ~20 MB file
 ```
 *NOTE*: Once the dumps are stored, please make sure to restart the python interpreter so that the changes take effect. 
-*NOTE*: If you experience API connection issues (`ConnectionError`), since v0.2.12 there are automatic retries which you can even control and raise from the default of 10, as in `biorxiv(max_retries=20)`, thanks to [@memray](https://github.com/memray) for contributions!
+*NOTE*: If you experience API connection issues (`ConnectionError`), since v0.2.12 there are automatic retries which you can even control and raise from the default of 10, as in `biorxiv(max_retries=20)`.
 
-Since v0.2.5 `paperscraper` also allows to scrape {med/bio/chem}rxiv for specific dates! Thanks to [@achouhan93 ](https://github.com/achouhan93 ) for contributions!
+Since v0.2.5 `paperscraper` also allows to scrape {med/bio/chem}rxiv for specific dates.
 ```py
 medrxiv(begin_date="2023-04-01", end_date="2023-04-08")
 ```
@@ -350,3 +350,12 @@ If you use `paperscraper`, please cite the papers that motivated our development
 	author = {Jannis Born and David Beymer and Deepta Rajan and Adam Coy and Vandana V. Mukherjee and Matteo Manica and Prasanth Prasanna and Deddeh Ballah and Michal Guindy and Dorith Shaham and Pallav L. Shah and Emmanouil Karteris and Jan L. Robertus and Maria Gabrani and Michal Rosen-Zvi}
 }
 ```
+
+## Contributions
+Thanks to the following contributors:
+- @memray: Since `v0.2.12` there are automatic retries when downloading the {med/bio/chem}rxiv dumps.
+- @achouhan93: Since `v0.2.5` {med/bio/chem}rxiv can be scraped for specific dates!
+- @daenuprobst: Since  `v0.2.4` PDF files can be scraped directly (`paperscraper.pdf.save_pdf`)
+- @oppih: Since `v0.2.3` chemRxiv API also provides DOI and URL if available
+- @lukasschwab: Bumped `arxiv` dependency to >`1.4.2` in paperscraper `v0.1.0`.
+- @juliusbierk: Bugfixes

{paperscraper-0.2.12 → paperscraper-0.2.13}/README.md

@@ -1,6 +1,6 @@
 
-[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml/badge.svg)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml)
-[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml/badge.svg)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml)
+[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml/badge.svg?branch=main)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml?query=branch%3Amain)
+[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml/badge.svg?branch=main)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml?query=branch%3Amain)
 [![License:
 MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![PyPI version](https://badge.fury.io/py/paperscraper.svg)](https://badge.fury.io/py/paperscraper)
@@ -37,9 +37,9 @@ biorxiv()  # Takes ~1h and should result in ~350 MB file
 chemrxiv()  #  Takes ~45min and should result in ~20 MB file
 ```
 *NOTE*: Once the dumps are stored, please make sure to restart the python interpreter so that the changes take effect. 
-*NOTE*: If you experience API connection issues (`ConnectionError`), since v0.2.12 there are automatic retries which you can even control and raise from the default of 10, as in `biorxiv(max_retries=20)`, thanks to [@memray](https://github.com/memray) for contributions!
+*NOTE*: If you experience API connection issues (`ConnectionError`), since v0.2.12 there are automatic retries which you can even control and raise from the default of 10, as in `biorxiv(max_retries=20)`.
 
-Since v0.2.5 `paperscraper` also allows to scrape {med/bio/chem}rxiv for specific dates! Thanks to [@achouhan93 ](https://github.com/achouhan93 ) for contributions!
+Since v0.2.5 `paperscraper` also allows to scrape {med/bio/chem}rxiv for specific dates.
 ```py
 medrxiv(begin_date="2023-04-01", end_date="2023-04-08")
 ```
@@ -317,3 +317,12 @@ If you use `paperscraper`, please cite the papers that motivated our development
 	author = {Jannis Born and David Beymer and Deepta Rajan and Adam Coy and Vandana V. Mukherjee and Matteo Manica and Prasanth Prasanna and Deddeh Ballah and Michal Guindy and Dorith Shaham and Pallav L. Shah and Emmanouil Karteris and Jan L. Robertus and Maria Gabrani and Michal Rosen-Zvi}
 }
 ```
+
+## Contributions
+Thanks to the following contributors:
+- @memray: Since `v0.2.12` there are automatic retries when downloading the {med/bio/chem}rxiv dumps.
+- @achouhan93: Since `v0.2.5` {med/bio/chem}rxiv can be scraped for specific dates!
+- @daenuprobst: Since  `v0.2.4` PDF files can be scraped directly (`paperscraper.pdf.save_pdf`)
+- @oppih: Since `v0.2.3` chemRxiv API also provides DOI and URL if available
+- @lukasschwab: Bumped `arxiv` dependency to >`1.4.2` in paperscraper `v0.1.0`.
+- @juliusbierk: Bugfixes

{paperscraper-0.2.12 → paperscraper-0.2.13}/paperscraper/__init__.py

@@ -1,7 +1,7 @@
 """Initialize the module."""
 
 __name__ = "paperscraper"
-__version__ = "0.2.12"
+__version__ = "0.2.13"
 
 import logging
 import os

{paperscraper-0.2.12 → paperscraper-0.2.13}/paperscraper/scholar/scholar.py

@@ -15,6 +15,7 @@ scholar_field_mapper = {
     "venue": "journal",
     "author": "authors",
     "cites": "citations",
+    "pub_year": "year",
 }
 process_fields = {"year": lambda x: int(x) if x.isdigit() else -1, "citations": int}
 
@@ -46,16 +47,21 @@ def get_scholar_papers(
 
     matches = scholarly.search_pubs(title)
 
-    processed = [
-        {
+    processed = []
+    for paper in matches:
+
+        # Extracts title, author, year, journal, abstract
+        entry = {
             scholar_field_mapper.get(key, key): process_fields.get(
                 scholar_field_mapper.get(key, key), lambda x: x
             )(value)
-            for key, value in paper.bib.items()
+            for key, value in paper["bib"].items()
             if scholar_field_mapper.get(key, key) in fields
         }
-        for paper in matches
-    ]
+
+        entry["citations"] = paper["num_citations"]
+        processed.append(entry)
+
     return pd.DataFrame(processed)
 
 
@@ -99,7 +105,7 @@ def get_citations_from_title(title: str) -> int:
     title = '"' + title.strip() + '"'
 
     matches = scholarly.search_pubs(title)
-    counts = list(map(lambda p: int(p.bib["cites"]), matches))
+    counts = list(map(lambda p: int(p["num_citations"]), matches))
     if len(counts) == 0:
         logger.warning(f"Found no match for {title}.")
         return 0

paperscraper-0.2.13/paperscraper/scholar/tests/test_scholar.py

@@ -0,0 +1,60 @@
+import logging
+import pandas as pd
+import pytest
+from scholarly._proxy_generator import MaxTriesExceededException
+import functools
+
+from paperscraper.scholar import (
+    get_and_dump_scholar_papers,
+    get_citations_from_title,
+    get_scholar_papers,
+)
+
+logging.disable(logging.INFO)
+
+
+def handle_scholar_exception(func):
+    @functools.wraps(func)
+    def wrapper(*args, **kwargs):
+        try:
+            return func(*args, **kwargs)
+        except MaxTriesExceededException as e:
+            logging.info(f"MaxTriesExceededException caught: {e}")
+            pytest.skip("Skipping test due to MaxTriesExceededException")
+
+    return wrapper
+
+
+class TestScholar:
+
+    @handle_scholar_exception
+    def test_citations(self):
+        num = get_citations_from_title("GT4SD")
+        assert isinstance(num, int)
+        assert num > 0
+
+    @handle_scholar_exception
+    def test_dump_search(self, tmpdir):
+        temp_dir = tmpdir.mkdir("scholar_papers")
+        output_filepath = temp_dir.join("results.jsonl")
+        get_and_dump_scholar_papers("GT4SD", str(output_filepath))
+        assert output_filepath.check(file=1)
+
+    @handle_scholar_exception
+    def test_basic_search(self):
+        results = get_scholar_papers("GT4SD")
+        assert len(results) > 0  # Ensure we get some results
+        assert isinstance(results, pd.DataFrame)
+        assert all(
+            [
+                x in results.columns
+                for x in [
+                    "title",
+                    "abstract",
+                    "citations",
+                    "year",
+                    "authors",
+                    "journal",
+                ]
+            ]
+        )

{paperscraper-0.2.12 → paperscraper-0.2.13}/paperscraper.egg-info/PKG-INFO

@@ -1,8 +1,8 @@
 Metadata-Version: 2.1
 Name: paperscraper
-Version: 0.2.12
+Version: 0.2.13
 Summary: paperscraper: Package to scrape papers.
-Home-page: https://github.com/PhosphorylatedRabbits/paperscraper
+Home-page: https://github.com/jannisborn/paperscraper
 Author: Jannis Born, Matteo Manica
 Author-email: jannis.born@gmx.de, drugilsberg@gmail.com
 License: MIT
@@ -22,7 +22,7 @@ Requires-Dist: pymed
 Requires-Dist: pandas
 Requires-Dist: requests
 Requires-Dist: tqdm
-Requires-Dist: scholarly==0.5.1
+Requires-Dist: scholarly>=1.0.0
 Requires-Dist: seaborn
 Requires-Dist: matplotlib
 Requires-Dist: matplotlib_venn
@@ -32,8 +32,8 @@ Requires-Dist: thefuzz
 Requires-Dist: pytest
 
 
-[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml/badge.svg)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml)
-[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml/badge.svg)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml)
+[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml/badge.svg?branch=main)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_tip.yml?query=branch%3Amain)
+[![build](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml/badge.svg?branch=main)](https://github.com/PhosphorylatedRabbits/paperscraper/actions/workflows/test_pypi.yml?query=branch%3Amain)
 [![License:
 MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![PyPI version](https://badge.fury.io/py/paperscraper.svg)](https://badge.fury.io/py/paperscraper)
@@ -70,9 +70,9 @@ biorxiv()  # Takes ~1h and should result in ~350 MB file
 chemrxiv()  #  Takes ~45min and should result in ~20 MB file
 ```
 *NOTE*: Once the dumps are stored, please make sure to restart the python interpreter so that the changes take effect. 
-*NOTE*: If you experience API connection issues (`ConnectionError`), since v0.2.12 there are automatic retries which you can even control and raise from the default of 10, as in `biorxiv(max_retries=20)`, thanks to [@memray](https://github.com/memray) for contributions!
+*NOTE*: If you experience API connection issues (`ConnectionError`), since v0.2.12 there are automatic retries which you can even control and raise from the default of 10, as in `biorxiv(max_retries=20)`.
 
-Since v0.2.5 `paperscraper` also allows to scrape {med/bio/chem}rxiv for specific dates! Thanks to [@achouhan93 ](https://github.com/achouhan93 ) for contributions!
+Since v0.2.5 `paperscraper` also allows to scrape {med/bio/chem}rxiv for specific dates.
 ```py
 medrxiv(begin_date="2023-04-01", end_date="2023-04-08")
 ```
@@ -350,3 +350,12 @@ If you use `paperscraper`, please cite the papers that motivated our development
 	author = {Jannis Born and David Beymer and Deepta Rajan and Adam Coy and Vandana V. Mukherjee and Matteo Manica and Prasanth Prasanna and Deddeh Ballah and Michal Guindy and Dorith Shaham and Pallav L. Shah and Emmanouil Karteris and Jan L. Robertus and Maria Gabrani and Michal Rosen-Zvi}
 }
 ```
+
+## Contributions
+Thanks to the following contributors:
+- @memray: Since `v0.2.12` there are automatic retries when downloading the {med/bio/chem}rxiv dumps.
+- @achouhan93: Since `v0.2.5` {med/bio/chem}rxiv can be scraped for specific dates!
+- @daenuprobst: Since  `v0.2.4` PDF files can be scraped directly (`paperscraper.pdf.save_pdf`)
+- @oppih: Since `v0.2.3` chemRxiv API also provides DOI and URL if available
+- @lukasschwab: Bumped `arxiv` dependency to >`1.4.2` in paperscraper `v0.1.0`.
+- @juliusbierk: Bugfixes

{paperscraper-0.2.12 → paperscraper-0.2.13}/paperscraper.egg-info/SOURCES.txt

@@ -30,6 +30,8 @@ paperscraper/pubmed/pubmed.py
 paperscraper/pubmed/utils.py
 paperscraper/scholar/__init__.py
 paperscraper/scholar/scholar.py
+paperscraper/scholar/tests/__init__.py
+paperscraper/scholar/tests/test_scholar.py
 paperscraper/server_dumps/__init__.py
 paperscraper/tests/__init__.py
 paperscraper/tests/test_dumpy.py

{paperscraper-0.2.12 → paperscraper-0.2.13}/paperscraper.egg-info/requires.txt

@@ -3,7 +3,7 @@ pymed
 pandas
 requests
 tqdm
-scholarly==0.5.1
+scholarly>=1.0.0
 seaborn
 matplotlib
 matplotlib_venn

{paperscraper-0.2.12 → paperscraper-0.2.13}/setup.py

@@ -1,4 +1,5 @@
 """Install package."""
+
 import io
 import os
 import re
@@ -23,7 +24,7 @@ setup(
     long_description_content_type="text/markdown",
     author="Jannis Born, Matteo Manica",
     author_email=("jannis.born@gmx.de, drugilsberg@gmail.com"),
-    url="https://github.com/PhosphorylatedRabbits/paperscraper",
+    url="https://github.com/jannisborn/paperscraper",
     license="MIT",
     install_requires=[
         "arxiv>=1.4.2",
@@ -31,7 +32,7 @@ setup(
         "pandas",
         "requests",
         "tqdm",
-        "scholarly==0.5.1",
+        "scholarly>=1.0.0",
         "seaborn",
         "matplotlib",
         "matplotlib_venn",