papers-cli 2.12.1__tar.gz → 2.14__tar.gz

papers_cli-2.14/.github/FUNDING.yml

@@ -0,0 +1,2 @@
+# These are supported funding model platforms
+github: [perrette]

{papers_cli-2.12.1 → papers_cli-2.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: papers-cli
-Version: 2.12.1
+Version: 2.14
 Summary: utilities to keep your PDF library organized
 Author-email: Mahé Perrette <mahe.perrette@gmail.com>
 License: MIT
@@ -17,7 +17,7 @@ Requires-Dist: scholarly
 Requires-Dist: rapidfuzz
 Requires-Dist: normality
 
-[![test](https://github.com/perrette/papers/workflows/CI/badge.svg?branch=main)](https://github.com/perrette/papers/actions)
+[![test](https://github.com/perrette/papers/workflows/CI/badge.svg?query=branch%3Amaster)](https://github.com/perrette/papers/actions)
 [![python](https://img.shields.io/badge/python-3.9-blue.svg)]()
 [![python](https://img.shields.io/badge/python-3.10-blue.svg)]()
 [![python](https://img.shields.io/badge/python-3.11-blue.svg)]()
@@ -306,6 +306,14 @@ That command is also convenient to check on what's actually tracked and what is
     # ...
     papers filecheck --rename --filesdir files
 
+There is also a command specifically designed to clean up the zombie files and folders:
+
+    papers filecheck --clean-filesdir
+
+That command will ask before removig anything, unless `--force` is passed. Currently
+it ignores hidden files and folders, and will only consider folder that have a `.{folder}.bib` file inside, 
+which is the convention `papers` follows to store multiple attachments. That command works best
+when the files are in their own folder, and not mixed up with other things, obviously.
 
 ### Setup git-tracked library (optional)
 
@@ -321,14 +329,8 @@ If `--git-lfs` is passed, the files will be backed-up along with the bibtex.
 Under the hood, bibtex and files (if applicable) are copied (hard-linked) to a back-up directory.
 Details are described in [issue 51](https://github.com/perrette/papers/issues/51).
 
-In local installs, backup occurs in `.papers/`. In global installs, defaults to `~/.local/.share/papers`.
-Type `papers status -v` to find out.
-
+Backup occurs in a subfolder of `~/.local/.share/papers` regardless of the type of installation. Type `papers status -v` to find out.
 For local install that are already git-tracked, the feature remains useful as it is the basis for `papers undo` and `papers redo`.
-You might want to add `.papers` to your .gitignore to avoid messing up with your larger project.
-
-This feature is experimental.
-
 
 ### undo / redo
 

{papers_cli-2.12.1 → papers_cli-2.14}/README.md

@@ -1,4 +1,4 @@
-[![test](https://github.com/perrette/papers/workflows/CI/badge.svg?branch=main)](https://github.com/perrette/papers/actions)
+[![test](https://github.com/perrette/papers/workflows/CI/badge.svg?query=branch%3Amaster)](https://github.com/perrette/papers/actions)
 [![python](https://img.shields.io/badge/python-3.9-blue.svg)]()
 [![python](https://img.shields.io/badge/python-3.10-blue.svg)]()
 [![python](https://img.shields.io/badge/python-3.11-blue.svg)]()
@@ -287,6 +287,14 @@ That command is also convenient to check on what's actually tracked and what is
     # ...
     papers filecheck --rename --filesdir files
 
+There is also a command specifically designed to clean up the zombie files and folders:
+
+    papers filecheck --clean-filesdir
+
+That command will ask before removig anything, unless `--force` is passed. Currently
+it ignores hidden files and folders, and will only consider folder that have a `.{folder}.bib` file inside, 
+which is the convention `papers` follows to store multiple attachments. That command works best
+when the files are in their own folder, and not mixed up with other things, obviously.
 
 ### Setup git-tracked library (optional)
 
@@ -302,14 +310,8 @@ If `--git-lfs` is passed, the files will be backed-up along with the bibtex.
 Under the hood, bibtex and files (if applicable) are copied (hard-linked) to a back-up directory.
 Details are described in [issue 51](https://github.com/perrette/papers/issues/51).
 
-In local installs, backup occurs in `.papers/`. In global installs, defaults to `~/.local/.share/papers`.
-Type `papers status -v` to find out.
-
+Backup occurs in a subfolder of `~/.local/.share/papers` regardless of the type of installation. Type `papers status -v` to find out.
 For local install that are already git-tracked, the feature remains useful as it is the basis for `papers undo` and `papers redo`.
-You might want to add `.papers` to your .gitignore to avoid messing up with your larger project.
-
-This feature is experimental.
-
 
 ### undo / redo
 

{papers_cli-2.12.1 → papers_cli-2.14}/papers/__main__.py

@@ -9,17 +9,16 @@ import subprocess as sp
 import shutil
 import itertools
 import fnmatch   # unix-like match
-import hashlib
-import time
 
 import papers
 from papers import logger
 from papers.extract import extract_pdf_doi, isvaliddoi, extract_pdf_metadata
-from papers.extract import fetch_bibtex_by_doi
+from papers.extract import fetch_bibtex_by_doi, fetch_bibtex_by_fulltext_crossref, fetch_bibtex_by_fulltext_scholar
 from papers.encoding import parse_file, format_file, family_names, format_entries, standard_name, format_entry, parse_keywords, format_key
 from papers.config import bcolors, Config, search_config, CONFIG_FILE, CONFIG_FILE_LOCAL, DATA_DIR, CONFIG_FILE_LEGACY, BACKUP_DIR
 from papers.duplicate import list_duplicates, list_uniques, edit_entries
-from papers.bib import Biblio, FUZZY_RATIO, DEFAULT_SIMILARITY, entry_filecheck, backupfile as backupfile_func, isvalidkey, DuplicateKeyError
+from papers.bib import (Biblio, FUZZY_RATIO, DEFAULT_SIMILARITY, entry_filecheck,
+                        backupfile as backupfile_func, isvalidkey, DuplicateKeyError, clean_filesdir)
 from papers.utils import move, checksum, view_pdf, open_folder
 from papers import __version__
 
@@ -89,6 +88,9 @@ def _backup_bib(biblio, config, message=None):
         biblio.rename_entries_files(copy=True, relative_to=backupdir, hardlink=True)
         biblio.save(config.backupfile_clean)
         config.gitcmd(f"add {config.backupfile_clean.name}")
+
+        # Remove unlink files
+        clean_filesdir(biblio, interactive=False, ignore_files=[config.backupfile, config.backupfile_clean])
         config.gitcmd(f"add files")
 
     else:
@@ -288,19 +290,41 @@ def installcmd(parser, o, config):
     """
     prompt = o.prompt and not o.edit
     # installed = config.file is not None
+    # WARNING if pre-existing config file
     if config.file is not None and prompt:
-        while True:
-            ans = input(f'An existing {"local" if config.local else "global"} install was found: {config.file}. Overwrite (O) or Edit (E) ? [o / e]')
-            if ans.lower() in ('o', 'e'):
-                break
-            else:
-                print('Use the --edit option to selectively edit existing configuration, or --force to ignore pre-existing configuration.')
-        o.edit = ans.lower() == 'e'
+        if ((o.local and config.local) or ((not o.local and not config.local))):
+            while True:
+                ans = input(f'An existing {"local" if config.local else "global"} install was found: {config.file}. Overwrite (O) or Edit (E) ? [o / e]')
+                if ans.lower() in ('o', 'e'):
+                    break
+                else:
+                    print('Use the --edit option to selectively edit existing configuration, or --force to ignore pre-existing configuration.')
+            o.edit = ans.lower() == 'e'
+
+        elif not o.local and config.local:
+            print(f"A local configuration file was found that will have precedence over the global install: {config.file}. Delete (D) or Ignore (any other key) ?")
+            ans = input()
+            if ans.lower() == "d":
+                logger.info(f"Removing pre-existing local configuration file {config.file}")
+                os.remove(config.file)
+                config = Config()
+
+        elif o.local and not config.local:
+            logger.warning(f"A global configuration file was found, but the local install will have precedence: {config.file}. Delete the global configuration file (D) or Ignore (any other key) ?")
+            ans = input()
+            if ans.lower() == "d":
+                logger.info(f"Removing pre-existing global configuration file {config.file}")
+                os.remove(config.file)
+                config = Config()
+
+        else:
+            raise PapersExit("Unexpected configuration state")
+
 
     if not o.edit:
-        if config.local and os.path.exists(config.file):
+        if config.file and os.path.exists(config.file) and ((o.local and config.local) or ((not o.local and not config.local))):
             # if we don't do that the local install will take precedence over global install
-            logger.warning(f"remove pre-existing local configuration file: {config.file}")
+            logger.warning(f"remove pre-existing configuration file: {config.file}")
             os.remove(config.file)
         config = Config()
 
@@ -673,6 +697,10 @@ def filecheckcmd(parser, o, config):
     if o.rename:
         biblio.rename_entries_files(o.copy)
 
+    if o.clean_filesdir:
+        print("Check files directory for unlinked files")
+        clean_filesdir(biblio, interactive=not o.force, ignore_files=[config.bibtex])
+
     savebib(biblio, config)
 
 def redocmd(parser, o, config):
@@ -717,7 +745,20 @@ def doicmd(parser, o):
     print(extract_pdf_doi(o.pdf, image=o.image))
 
 def fetchcmd(parser, o):
-    print(fetch_bibtex_by_doi(o.doi))
+    # either one or several DOIs
+    if all(isvaliddoi(field) for field in o.doi_or_text):
+        if o.scholar:
+            parser.error("Fetching from DOI does not support Google Scholar option")
+        for doi in o.doi_or_text:
+            print(fetch_bibtex_by_doi(doi))
+        return
+
+    # or one full text search
+    field = " ".join(o.doi_or_text)
+    if o.scholar:
+        print(fetch_bibtex_by_fulltext_scholar(field))
+    else:
+        print(fetch_bibtex_by_fulltext_crossref(field))
 
 def extractcmd(parser, o):
     print(extract_pdf_metadata(o.pdf, search_doi=not o.fulltext, search_fulltext=True, scholar=o.scholar, minwords=o.word_count, max_query_words=o.word_count, image=o.image))
@@ -1103,6 +1144,9 @@ def get_parser(config=None):
     filecheckp.add_argument('-d', '--delete-broken', action='store_true',
         help='remove file entry if the file link is broken')
 
+    filecheckp.add_argument('--clean-filesdir', action='store_true',
+        help='remove files in filesdir if not referred to in any entry')
+
     filecheckp.add_argument('--fix-mendeley', action='store_true',
         help='fix a Mendeley bug where the leading "/" is omitted.')
 
@@ -1174,15 +1218,15 @@ def get_parser(config=None):
 
     # doi
     # ===
-    doip = subparsers.add_parser('doi', description='parse DOI from PDF')
+    doip = subparsers.add_parser('doi', description='parse DOI from PDF', parents=[loggingp])
     doip.add_argument('pdf')
     doip.add_argument('--image', action='store_true', help='convert to image and use tesseract instead of pdftotext')
 
     # fetch
     # =====
-    fetchp = subparsers.add_parser('fetch', description='fetch bibtex from DOI')
-    fetchp.add_argument('doi')
-
+    fetchp = subparsers.add_parser('fetch', description='fetch bibtex from DOI or full-text', parents=[loggingp])
+    fetchp.add_argument('doi_or_text', nargs='+', help='DOI or full text.')
+    fetchp.add_argument('--scholar', action='store_true', help='use google scholar instead of default crossref for fulltext search')
 
     # extract
     # ========
@@ -1219,6 +1263,13 @@ def get_parser(config=None):
 # Main script
 #############
 
+def handle_logging(o):
+    # verbosity
+    if getattr(o,'logging_level',None):
+        logger.setLevel(o.logging_level)
+    logger.debug("LOGGER LEVEL: "+logging.getLevelName(logger.getEffectiveLevel()))
+
+
 def main(args=None):
     papers.config.DRYRUN = False  # reset in case main() if called directly
     if args is not None:
@@ -1242,9 +1293,8 @@ def main(args=None):
         print(__version__)
         return
 
-    # verbosity
-    if getattr(o,'logging_level',None):
-        logger.setLevel(o.logging_level)
+    handle_logging(o)
+
     # modify disk state?
     if hasattr(o,'dry_run'):
         papers.config.DRYRUN = o.dry_run

{papers_cli-2.12.1 → papers_cli-2.14}/papers/_version.py

@@ -1,8 +1,13 @@
-# file generated by setuptools_scm
+# file generated by setuptools-scm
 # don't change, don't track in version control
+
+__all__ = ["__version__", "__version_tuple__", "version", "version_tuple"]
+
 TYPE_CHECKING = False
 if TYPE_CHECKING:
-    from typing import Tuple, Union
+    from typing import Tuple
+    from typing import Union
+
     VERSION_TUPLE = Tuple[Union[int, str], ...]
 else:
     VERSION_TUPLE = object
@@ -12,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '2.12.1'
-__version_tuple__ = version_tuple = (2, 12, 1)
+__version__ = version = '2.14'
+__version_tuple__ = version_tuple = (2, 14)

{papers_cli-2.12.1 → papers_cli-2.14}/papers/bib.py

@@ -1,13 +1,8 @@
 import os
 from pathlib import Path
-import logging
-# logger.basicConfig(level=logger.INFO)
-import argparse
-import subprocess as sp
 import shutil
 import bisect
-import itertools
-import fnmatch   # unix-like match
+from unidecode import unidecode as unicode_to_ascii
 import bibtexparser
 from bibtexparser.customization import convert_to_unicode
 
@@ -18,8 +13,8 @@ from papers.extract import extract_pdf_doi, isvaliddoi, parse_doi
 from papers.extract import extract_pdf_metadata
 from papers.extract import fetch_bibtex_by_fulltext_crossref, fetch_bibtex_by_doi
 
-from papers.encoding import unicode_to_latex, unicode_to_ascii
 from papers.encoding import parse_file, format_file, standard_name, family_names, format_entries, update_file_path, format_entry
+from papers.latexenc import unicode_to_latex, latex_to_unicode
 
 from papers.filename import NAMEFORMAT, KEYFORMAT
 from papers.utils import bcolors, checksum, move as _move
@@ -309,7 +304,7 @@ class Biblio:
             else:
                 print(format_entry(self, entry, prefix="New entry"))
                 print(format_entry(self, self.entries[i], prefix="Conflicts with"))
-                raise DuplicateKeyError('Key conflict. Try update_key is True?')
+                raise DuplicateKeyError('Key conflict. Try update_key is True or specify --key KEY explicitly')
 
         else:
             logger.info('new entry: '+self.key(entry))
@@ -821,3 +816,71 @@ def entry_filecheck(e, delete_broken=False, fix_mendeley=False,
         newfiles.append(file)
 
     e['file'] = format_file(newfiles, relative_to=relative_to)
+
+
+def clean_filesdir(biblio, interactive=True, ignore_files=None, ignore_folders=None):
+    if biblio.filesdir is None:
+        raise ValueError('filesdir is not defined, cannot clean')
+    removed_files = []
+
+    allfiles = set(os.path.abspath(file) for e in biblio.entries for file in biblio.get_files(e))
+    if ignore_files:
+        for f in ignore_files:
+            allfiles.add(os.path.abspath(f))
+    allfolders = set(os.path.abspath(folder) for e in biblio.entries for folder in {os.path.dirname(file) for file in biblio.get_files(e)})
+
+    if ignore_folders is None:
+        ignore_folders = []
+
+    for root, direcs, files in os.walk(biblio.filesdir):
+
+        ANS = None  # Y(es) or N(o) for all files in this folder (reset at each pass)
+        if root in ignore_folders:
+            continue
+        if '.git' in root.split(os.path.sep):
+            continue
+        for file in files:
+            path = os.path.abspath(os.path.join(root, file))
+            if file.startswith('.') or file.endswith('.bib'):
+                continue
+            if path not in allfiles:
+                if interactive:
+                    if ANS is None:
+                        ans = input(f"File: {os.path.relpath(path, biblio.filesdir)} not in library. Remove ? [Y/n/Y/N] ")
+                    else:
+                        ans = ANS
+                    if ans in ['Y', 'N']:
+                        ans = ANS = ans.lower()  # same reply for all files in this folder
+                    if ans.lower() == 'y':
+                        os.remove(path)
+                        removed_files.append(path)
+                else:
+                    logger.info(f"Remove unlinked file {os.path.relpath(path, biblio.filesdir)}.")
+                    os.remove(path)
+                    removed_files.append(path)
+
+        for direc in direcs:
+            if direc.startswith('.'):
+                continue
+            if direc in ignore_folders:
+                continue
+
+            # Check multifile entries
+            bibtex = os.path.join(root, direc, '.'+direc+'.bib')
+            if not os.path.exists(bibtex):
+                continue
+
+            direcpath = os.path.abspath(os.path.join(root, direc))
+
+            if direcpath not in allfolders:
+                if interactive:
+                    ans = input(f"Folder: {os.path.relpath(direcpath, root)} not in library. Remove ? [Y/n] ")
+                    if ans.lower() == 'y':
+                        shutil.rmtree(os.path.join(root, direcpath))
+                else:
+                    logger.info(f"Remove folder {direc}.")
+                    shutil.rmtree(os.path.join(root, direcpath))
+
+        break
+
+    return removed_files

{papers_cli-2.12.1 → papers_cli-2.14}/papers/config.py

@@ -85,7 +85,7 @@ class Config:
             return Path(os.path.sep)
 
     def gitcmd(self, cmd, check=True, **kw):
-        logger.debug(f"git add {self.backupfile.name}")
+        logger.debug(f"git {cmd} -C {self.gitdir}")
         try:
             sp.run(
                 f"git {cmd}",

{papers_cli-2.12.1 → papers_cli-2.14}/papers/encoding.py

@@ -2,7 +2,6 @@ import os
 import bibtexparser
 from pathlib import Path
 from unidecode import unidecode as unicode_to_ascii
-from papers.latexenc import unicode_to_latex
 from papers.utils import ansi_link as link, bcolors
 from papers import logger
 

{papers_cli-2.12.1 → papers_cli-2.14}/papers_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: papers-cli
-Version: 2.12.1
+Version: 2.14
 Summary: utilities to keep your PDF library organized
 Author-email: Mahé Perrette <mahe.perrette@gmail.com>
 License: MIT
@@ -17,7 +17,7 @@ Requires-Dist: scholarly
 Requires-Dist: rapidfuzz
 Requires-Dist: normality
 
-[![test](https://github.com/perrette/papers/workflows/CI/badge.svg?branch=main)](https://github.com/perrette/papers/actions)
+[![test](https://github.com/perrette/papers/workflows/CI/badge.svg?query=branch%3Amaster)](https://github.com/perrette/papers/actions)
 [![python](https://img.shields.io/badge/python-3.9-blue.svg)]()
 [![python](https://img.shields.io/badge/python-3.10-blue.svg)]()
 [![python](https://img.shields.io/badge/python-3.11-blue.svg)]()
@@ -306,6 +306,14 @@ That command is also convenient to check on what's actually tracked and what is
     # ...
     papers filecheck --rename --filesdir files
 
+There is also a command specifically designed to clean up the zombie files and folders:
+
+    papers filecheck --clean-filesdir
+
+That command will ask before removig anything, unless `--force` is passed. Currently
+it ignores hidden files and folders, and will only consider folder that have a `.{folder}.bib` file inside, 
+which is the convention `papers` follows to store multiple attachments. That command works best
+when the files are in their own folder, and not mixed up with other things, obviously.
 
 ### Setup git-tracked library (optional)
 
@@ -321,14 +329,8 @@ If `--git-lfs` is passed, the files will be backed-up along with the bibtex.
 Under the hood, bibtex and files (if applicable) are copied (hard-linked) to a back-up directory.
 Details are described in [issue 51](https://github.com/perrette/papers/issues/51).
 
-In local installs, backup occurs in `.papers/`. In global installs, defaults to `~/.local/.share/papers`.
-Type `papers status -v` to find out.
-
+Backup occurs in a subfolder of `~/.local/.share/papers` regardless of the type of installation. Type `papers status -v` to find out.
 For local install that are already git-tracked, the feature remains useful as it is the basis for `papers undo` and `papers redo`.
-You might want to add `.papers` to your .gitignore to avoid messing up with your larger project.
-
-This feature is experimental.
-
 
 ### undo / redo
 

{papers_cli-2.12.1 → papers_cli-2.14}/papers_cli.egg-info/SOURCES.txt

@@ -5,6 +5,7 @@ README.md
 pyproject.toml
 requirements.txt
 setup.py
+.github/FUNDING.yml
 .github/workflows/pypi.yml
 .github/workflows/tests.yml
 papers/__init__.py