scholarcli 1.26__tar.gz → 1.28__tar.gz

{scholarcli-1.26/src/scholarcli.egg-info → scholarcli-1.28}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scholarcli
-Version: 1.26
+Version: 1.28
 Summary: A tool for structured literature searches across bibliographic databases
 Author-email: Daniel Bosk <dbosk@kth.se>, Ric Glassey <glassey@kth.se>
 License-Expression: MIT
@@ -14,6 +14,7 @@ Requires-Dist: pyalex>=0.19
 Requires-Dist: arxiv>=2.1.0
 Requires-Dist: cachetools>=6.2.4
 Requires-Dist: platformdirs>=4.5.1
+Requires-Dist: filelock>=3.12
 Requires-Dist: textual>=6.11.0
 Requires-Dist: pypandoc>=1.14
 Requires-Dist: click>=8.0.0
@@ -36,6 +37,8 @@ Requires-Dist: pyyaml>=6.0
 Requires-Dist: questionary>=2.0.0
 Requires-Dist: llm-github-copilot>=0.3.1
 Requires-Dist: llm-mistral<0.16,>=0.15
+Requires-Dist: llm-openai-via-codex>=0.1a0
+Requires-Dist: pytorial>=0.8.1
 Dynamic: license-file
 
 # Scholar
@@ -160,7 +163,8 @@ Search results are cached to avoid redundant API calls:
 
 ```bash
 scholar cache info    # Show cache statistics
-scholar cache clear   # Delete cached results
+scholar cache clear   # Delete cached results (keeps paid LLM output;
+                      # add --include-llm to delete that too)
 scholar cache path    # Print cache directory
 ```
 
@@ -227,6 +231,30 @@ scholar search "machine learning privacy"
 scholar rq "How do LLMs support novice programming?" --count 20
 ```
 
+### Tutorials
+
+Scholar ships interactive tutorials (built on
+[pytorial](https://github.com/dbosk/pytorial)): each step opens a shell,
+you type the command the step asks for, and the transcript is checked
+before the lesson moves on.
+
+```bash
+scholar tutorial list                 # the lessons and your progress
+scholar tutorial run first-search     # start or resume a lesson
+scholar tutorial run --restart first-search
+scholar tutorial review first-search  # read back what you typed
+```
+
+The lessons, in order: `first-search` (search, providers, output formats,
+saving a session), `reviewing-in-the-tui` (recording decisions),
+`enrich-verify-export` (abstracts and open access, retractions, CSV and
+LaTeX exports), `llm-synthesis` (setting up the `llm` command, LLM
+classification, a literature overview) and `scholar-with-an-agent` (what
+an agent running the `backing-claims` skill does with Scholar, and how to
+audit it). Every lesson runs in its own workspace with its own session
+store, so practice sessions never appear in your real `scholar sessions
+list`.
+
 ## Installation
 
 If you don't use `pipx`, you can install with `pip`:
@@ -270,7 +298,7 @@ Some providers require API keys set as environment variables:
 | SSRN, bioRxiv, medRxiv, ChemRxiv, Research Square, Preprints.org, IACR | `SCHOLAR_EMAIL` | No | Any email (they search via OpenAlex) |
 | Web of Science | `WOS_EXPANDED_API_KEY` or `WOS_STARTER_API_KEY` | Yes | [developer.clarivate.com](https://developer.clarivate.com) |
 | IEEE Xplore | `IEEE_API_KEY` | Yes | [developer.ieee.org](https://developer.ieee.org) |
-| Scopus | `SCOPUS_API_KEY` | Yes | [dev.elsevier.com](https://dev.elsevier.com) |
+| Scopus | `SCOPUS_API_KEY` | Yes | [dev.elsevier.com](https://dev.elsevier.com). Searching works from any network, but abstracts, keywords and full author lists come only with Scopus's COMPLETE view, which is bound to the institution's IP range (or `SCOPUS_INST_TOKEN`). scholar asks for it and falls back by itself (`scholar providers check` tells which view you get); `SCOPUS_VIEW=standard` or `complete` overrides. A 400 "Error translating query" on *every* query is an Elsevier-side refusal, not a query error: it has cleared by itself before, so retry later |
 
 View provider status:
 
@@ -278,6 +306,38 @@ View provider status:
 scholar providers
 ```
 
+### Network: proxy and User-Agent
+
+To reach subscriber-only content from off campus (Scopus's COMPLETE
+view, a publisher that admits the institution's IP range), route
+scholar's requests through a proxy on that network, for example an SSH
+SOCKS tunnel:
+
+```bash
+ssh -D 127.0.0.1:1081 -N you@login.your-university.example &
+scholar --proxy socks5h://127.0.0.1:1081 search "..."
+export SCHOLAR_PROXY=socks5h://127.0.0.1:1081   # or for every command
+scholar providers check --show-ip                # route, exit IP, health
+```
+
+`--proxy URL` (a global option, before the command) wins over
+`SCHOLAR_PROXY`; with neither, the standard `HTTP(S)_PROXY`/`ALL_PROXY`
+variables apply as usual. Use `socks5h://` rather than `socks5://` so
+host names are resolved by the proxy, on the far side of the tunnel.
+SOCKS needs the PySocks package (`pipx inject scholarcli PySocks`);
+scholar refuses a SOCKS proxy without it rather than failing every
+request. `scholar providers check` always prints the route; `--show-ip`
+also asks api.ipify.org for the exit address (a third-party request, so
+only on demand).
+
+Requests identify themselves as
+`scholar/<version> (+https://github.com/dbosk/scholar)`. Your
+`SCHOLAR_EMAIL` is not in it, so publishers never see the address; only
+the services that ask for one (OpenAlex, Crossref, Unpaywall) get it,
+through their own parameters. Set `SCHOLAR_USER_AGENT` to replace it. Scholar does not imitate a browser: a Cloudflare challenge
+or a login page from a publisher means the PDF is not available to
+scripts, and scholar stops there.
+
 ## Usage Examples
 
 ### Basic Search

{scholarcli-1.26 → scholarcli-1.28}/README.md

@@ -120,7 +120,8 @@ Search results are cached to avoid redundant API calls:
 
 ```bash
 scholar cache info    # Show cache statistics
-scholar cache clear   # Delete cached results
+scholar cache clear   # Delete cached results (keeps paid LLM output;
+                      # add --include-llm to delete that too)
 scholar cache path    # Print cache directory
 ```
 
@@ -187,6 +188,30 @@ scholar search "machine learning privacy"
 scholar rq "How do LLMs support novice programming?" --count 20
 ```
 
+### Tutorials
+
+Scholar ships interactive tutorials (built on
+[pytorial](https://github.com/dbosk/pytorial)): each step opens a shell,
+you type the command the step asks for, and the transcript is checked
+before the lesson moves on.
+
+```bash
+scholar tutorial list                 # the lessons and your progress
+scholar tutorial run first-search     # start or resume a lesson
+scholar tutorial run --restart first-search
+scholar tutorial review first-search  # read back what you typed
+```
+
+The lessons, in order: `first-search` (search, providers, output formats,
+saving a session), `reviewing-in-the-tui` (recording decisions),
+`enrich-verify-export` (abstracts and open access, retractions, CSV and
+LaTeX exports), `llm-synthesis` (setting up the `llm` command, LLM
+classification, a literature overview) and `scholar-with-an-agent` (what
+an agent running the `backing-claims` skill does with Scholar, and how to
+audit it). Every lesson runs in its own workspace with its own session
+store, so practice sessions never appear in your real `scholar sessions
+list`.
+
 ## Installation
 
 If you don't use `pipx`, you can install with `pip`:
@@ -230,7 +255,7 @@ Some providers require API keys set as environment variables:
 | SSRN, bioRxiv, medRxiv, ChemRxiv, Research Square, Preprints.org, IACR | `SCHOLAR_EMAIL` | No | Any email (they search via OpenAlex) |
 | Web of Science | `WOS_EXPANDED_API_KEY` or `WOS_STARTER_API_KEY` | Yes | [developer.clarivate.com](https://developer.clarivate.com) |
 | IEEE Xplore | `IEEE_API_KEY` | Yes | [developer.ieee.org](https://developer.ieee.org) |
-| Scopus | `SCOPUS_API_KEY` | Yes | [dev.elsevier.com](https://dev.elsevier.com) |
+| Scopus | `SCOPUS_API_KEY` | Yes | [dev.elsevier.com](https://dev.elsevier.com). Searching works from any network, but abstracts, keywords and full author lists come only with Scopus's COMPLETE view, which is bound to the institution's IP range (or `SCOPUS_INST_TOKEN`). scholar asks for it and falls back by itself (`scholar providers check` tells which view you get); `SCOPUS_VIEW=standard` or `complete` overrides. A 400 "Error translating query" on *every* query is an Elsevier-side refusal, not a query error: it has cleared by itself before, so retry later |
 
 View provider status:
 
@@ -238,6 +263,38 @@ View provider status:
 scholar providers
 ```
 
+### Network: proxy and User-Agent
+
+To reach subscriber-only content from off campus (Scopus's COMPLETE
+view, a publisher that admits the institution's IP range), route
+scholar's requests through a proxy on that network, for example an SSH
+SOCKS tunnel:
+
+```bash
+ssh -D 127.0.0.1:1081 -N you@login.your-university.example &
+scholar --proxy socks5h://127.0.0.1:1081 search "..."
+export SCHOLAR_PROXY=socks5h://127.0.0.1:1081   # or for every command
+scholar providers check --show-ip                # route, exit IP, health
+```
+
+`--proxy URL` (a global option, before the command) wins over
+`SCHOLAR_PROXY`; with neither, the standard `HTTP(S)_PROXY`/`ALL_PROXY`
+variables apply as usual. Use `socks5h://` rather than `socks5://` so
+host names are resolved by the proxy, on the far side of the tunnel.
+SOCKS needs the PySocks package (`pipx inject scholarcli PySocks`);
+scholar refuses a SOCKS proxy without it rather than failing every
+request. `scholar providers check` always prints the route; `--show-ip`
+also asks api.ipify.org for the exit address (a third-party request, so
+only on demand).
+
+Requests identify themselves as
+`scholar/<version> (+https://github.com/dbosk/scholar)`. Your
+`SCHOLAR_EMAIL` is not in it, so publishers never see the address; only
+the services that ask for one (OpenAlex, Crossref, Unpaywall) get it,
+through their own parameters. Set `SCHOLAR_USER_AGENT` to replace it. Scholar does not imitate a browser: a Cloudflare challenge
+or a login page from a publisher means the PDF is not available to
+scripts, and scholar stops there.
+
 ## Usage Examples
 
 ### Basic Search

{scholarcli-1.26 → scholarcli-1.28}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "scholarcli"
-version = "1.26"
+version = "1.28"
 description = "A tool for structured literature searches across bibliographic databases"
 authors = [{ name = "Daniel Bosk", email = "dbosk@kth.se" },
            { name = "Ric Glassey", email = "glassey@kth.se" }]
@@ -15,6 +15,7 @@ dependencies = [
     "arxiv>=2.1.0",
     "cachetools>=6.2.4",
     "platformdirs>=4.5.1",
+    "filelock>=3.12",
     "textual>=6.11.0",
     "pypandoc>=1.14",
     "click>=8.0.0",
@@ -39,6 +40,8 @@ dependencies = [
     "questionary>=2.0.0",
     "llm-github-copilot>=0.3.1",
     "llm-mistral (>=0.15,<0.16)",
+    "llm-openai-via-codex>=0.1a0",
+    "pytorial>=0.8.1",
 ]
 
 [project.scripts]
@@ -51,6 +54,9 @@ build-backend = "setuptools.build_meta"
 [tool.setuptools.packages.find]
 where = ["src"]
 
+[tool.setuptools.package-data]
+scholar = ["tutorials/*.md"]
+
 [tool.pytest.ini_options]
 testpaths = ["tests"]
 python_files = ["test_*.py"]

{scholarcli-1.26 → scholarcli-1.28}/src/scholar/__init__.py

@@ -6,9 +6,15 @@ from .scholar import Search, SearchResult, Paper, SearchFilters
 from .scholar import search, filter_papers, deduplicate_papers
 from .scholar import deduplicate_versions, merge_version_of
 from .scholar import get_registry, isolated_registry
+from .scholar import hits_label, provider_outcome, result_outcome
+from .scholar import normalize_venue
 from .utils import safe_get_nested, ensure_list
 
 __all__ = [
+    "hits_label",
+    "provider_outcome",
+    "result_outcome",
+    "normalize_venue",
     "Search",
     "SearchResult",
     "Paper",

{scholarcli-1.26 → scholarcli-1.28}/src/scholar/cache.py

@@ -15,6 +15,7 @@ import pickle
 from typing import Any
 import atexit
 
+LLM_CACHE_PREFIX = "llm_"
 # Registry of caches to save on exit: provider_name -> cache dict
 CACHE_REGISTRY: dict[str, dict[Any, Any]] = {}
 
@@ -70,22 +71,49 @@ def save_cache(provider_name: str, cache: dict[Any, Any]) -> None:
         logger.warning(f"Failed to save cache for {provider_name}: {e}")
 
 
-def clear_cache() -> int:
+def is_llm_cache(name: str) -> bool:
+    """Return whether the cache `name` holds language-model output."""
+    return name.startswith(LLM_CACHE_PREFIX)
+
+
+def llm_cache_files() -> list[Path]:
+    """Return the LLM cache files in the cache directory."""
+    return sorted(
+        cache_file
+        for cache_file in get_cache_dir().glob("*.pkl")
+        if is_llm_cache(cache_file.stem)
+    )
+
+
+def clear_cache(include_llm: bool = False) -> int:
     """
     Clear all cached search results.
 
+    The LLM caches (see `is_llm_cache`) hold paid model output and are
+    kept unless `include_llm` is true.
+
+    Also empties the in-memory copy of every deleted cache that is
+    registered for saving at exit, so the exit hook cannot write the
+    old contents back.
+
     Returns the number of cache files deleted.
     """
     cache_dir = get_cache_dir()
     count = 0
     logger.info("Clearing all cache files")
     for cache_file in cache_dir.glob("*.pkl"):
+        if is_llm_cache(cache_file.stem) and not include_llm:
+            continue
         try:
             cache_file.unlink()
             count += 1
             logger.debug(f"Deleted cache file: {cache_file.name}")
         except OSError as e:
             logger.warning(f"Failed to delete {cache_file.name}: {e}")
+            continue
+        registered = CACHE_REGISTRY.get(cache_file.stem)
+        if registered is not None:
+            registered.clear()
     logger.info(f"Cleared {count} cache file(s)")
     return count
 
@@ -97,26 +125,31 @@ def get_cache_stats() -> dict[str, Any]:
     Returns a dictionary with:
     - cache_dir: Path to the cache directory
     - providers: Dict mapping provider names to entry counts
-    - total_entries: Total number of cached queries
-    - total_size_bytes: Total size of cache files
+      (search and lookup caches, which `clear_cache` removes)
+    - llm_caches: Dict mapping LLM cache names to entry counts
+      (kept by `clear_cache` unless `include_llm` is true)
+    - total_entries: Total number of cached entries in all files
+    - total_size_bytes: Total size of all cache files
     """
     cache_dir = get_cache_dir()
     providers: dict[str, int] = {}
+    llm_caches: dict[str, int] = {}
     total_size = 0
 
     for cache_file in cache_dir.glob("*.pkl"):
-        provider_name = cache_file.stem
+        name = cache_file.stem
         total_size += cache_file.stat().st_size
+        counts = llm_caches if is_llm_cache(name) else providers
         try:
-            cache = load_cache(provider_name)
-            providers[provider_name] = len(cache)
+            counts[name] = len(load_cache(name))
         except Exception:
-            providers[provider_name] = 0
+            counts[name] = 0
 
     return {
         "cache_dir": str(cache_dir),
         "providers": providers,
-        "total_entries": sum(providers.values()),
+        "llm_caches": llm_caches,
+        "total_entries": sum(providers.values()) + sum(llm_caches.values()),
         "total_size_bytes": total_size,
     }