scholarcli 1.24__tar.gz → 1.27__tar.gz

{scholarcli-1.24 → scholarcli-1.27}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scholarcli
-Version: 1.24
+Version: 1.27
 Summary: A tool for structured literature searches across bibliographic databases
 Author-email: Daniel Bosk <dbosk@kth.se>, Ric Glassey <glassey@kth.se>
 License-Expression: MIT
@@ -14,6 +14,7 @@ Requires-Dist: pyalex>=0.19
 Requires-Dist: arxiv>=2.1.0
 Requires-Dist: cachetools>=6.2.4
 Requires-Dist: platformdirs>=4.5.1
+Requires-Dist: filelock>=3.12
 Requires-Dist: textual>=6.11.0
 Requires-Dist: pypandoc>=1.14
 Requires-Dist: click>=8.0.0
@@ -36,6 +37,8 @@ Requires-Dist: pyyaml>=6.0
 Requires-Dist: questionary>=2.0.0
 Requires-Dist: llm-github-copilot>=0.3.1
 Requires-Dist: llm-mistral<0.16,>=0.15
+Requires-Dist: llm-openai-via-codex>=0.1a0
+Requires-Dist: pytorial>=0.8.1
 Dynamic: license-file
 
 # Scholar
@@ -46,19 +49,36 @@ A command-line tool for conducting structured literature searches across multipl
 
 ### Multi-Database Search
 
-Search across six academic databases with a single query:
+Search across fifteen academic databases with a single query:
 
 - **Semantic Scholar** - AI-powered research database with 200M+ papers
 - **OpenAlex** - Open catalog of 250M+ scholarly works
 - **DBLP** - Computer science bibliography
 - **Web of Science** - Comprehensive citation index (requires API key)
 - **IEEE Xplore** - IEEE technical literature (requires API key)
-- **arXiv** - Preprints (no API key)
+- **Scopus** - Elsevier's abstract and citation database (requires API key)
+- **arXiv** - Preprints in physics, math, CS (no API key)
+- **HAL** - The French national open archive (no API key)
+- **Preprint servers** - SSRN, bioRxiv, medRxiv, ChemRxiv, Research
+  Square, Preprints.org, and the IACR Cryptology ePrint Archive, each
+  searched through OpenAlex pinned to that server (none of them offers
+  a usable search API of its own)
+
+The preprint-server providers are opt-in: a default search already finds
+their papers through OpenAlex, so they run only when you name them —
+either individually (`-p ssrn` scopes a search to SSRN) or all at once
+with the `preprints` group.
 
 ```bash
 # Search specific providers
 scholar search "federated learning" -p s2 -p openalex
 
+# Scope a search to one preprint server
+scholar search "corporate governance" -p ssrn
+
+# All preprint servers at once (arXiv, HAL, SSRN, bioRxiv, medRxiv, ...)
+scholar search "diffusion models" -p preprints
+
 # Start from a research question (LLM generates provider-specific queries)
 scholar rq "How can privacy-preserving ML be evaluated?" \
   --provider openalex --provider dblp \
@@ -109,6 +129,16 @@ scholar sessions resume "machine learning"
 
 # Export session to reports
 scholar sessions export "machine learning" -f all
+
+# LaTeX report as an \input-able fragment (writes review.tex + review.bib)
+scholar sessions export "machine learning" -f latex --no-standalone -o review
+
+# Audit table for a backing appendix (one row per paper with the reason;
+# always a fragment; --lang en|sv, --theme TAG=NAME gives themes readable names)
+scholar sessions export "machine learning" -f table --lang sv --label sok-A \
+    --track "sökspår 1" --theme ml=maskininlärning -o review-full
+# ... listing only the cited/supporting/qualifying rows, the rest as counts
+scholar sessions export "machine learning" -f table --bearing-only -o review-bearing
 ```
 
 ### Paper Notes
@@ -133,12 +163,25 @@ Search results are cached to avoid redundant API calls:
 
 ```bash
 scholar cache info    # Show cache statistics
-scholar cache clear   # Delete cached results
+scholar cache clear   # Delete cached results (keeps paid LLM output;
+                      # add --include-llm to delete that too)
 scholar cache path    # Print cache directory
 ```
 
 PDF downloads are also cached for offline viewing.
 
+### Rate-Limit and Quota Tracking
+
+Provider rate limits and daily quotas (e.g. IEEE's 200 requests/day) are
+tracked persistently across invocations: a provider whose budget is spent
+is skipped with a warning naming the reset time instead of firing doomed
+requests.
+
+```bash
+scholar providers limits              # Show usage, remaining budget, reset times
+scholar providers limits --reset ieee # Clear tracked state for one provider
+```
+
 ## Quickstart
 
 ### Install
@@ -188,6 +231,30 @@ scholar search "machine learning privacy"
 scholar rq "How do LLMs support novice programming?" --count 20
 ```
 
+### Tutorials
+
+Scholar ships interactive tutorials (built on
+[pytorial](https://github.com/dbosk/pytorial)): each step opens a shell,
+you type the command the step asks for, and the transcript is checked
+before the lesson moves on.
+
+```bash
+scholar tutorial list                 # the lessons and your progress
+scholar tutorial run first-search     # start or resume a lesson
+scholar tutorial run --restart first-search
+scholar tutorial review first-search  # read back what you typed
+```
+
+The lessons, in order: `first-search` (search, providers, output formats,
+saving a session), `reviewing-in-the-tui` (recording decisions),
+`enrich-verify-export` (abstracts and open access, retractions, CSV and
+LaTeX exports), `llm-synthesis` (setting up the `llm` command, LLM
+classification, a literature overview) and `scholar-with-an-agent` (what
+an agent running the `backing-claims` skill does with Scholar, and how to
+audit it). Every lesson runs in its own workspace with its own session
+store, so practice sessions never appear in your real `scholar sessions
+list`.
+
 ## Installation
 
 If you don't use `pipx`, you can install with `pip`:
@@ -204,15 +271,34 @@ uv pip install scholarcli
 
 ## Configuration
 
+Set `SCHOLAR_EMAIL` to a contact address: OpenAlex (and thus every
+preprint-server provider), Crossref, and the polite pools use it for
+faster responses, and Unpaywall (open-access lookup during
+`scholar enrich` and `scholar pdf open <doi>`) requires it. `OPENALEX_EMAIL`
+and `CROSSREF_MAILTO` are still honoured as per-service overrides.
+
+OpenAlex meters requests against a daily budget (a search costs $0.001,
+reset at midnight UTC). A free API key from
+[openalex.org/settings/api](https://openalex.org/settings/api) raises the
+budget tenfold; set it as `OPENALEX_API_KEY` and it is used by the
+`openalex` provider and every preprint-server provider. `scholar
+providers check` then shows how much of the day's budget is left, and a
+search that hits the spent budget reports the reset time instead of a
+bare 429.
+
 Some providers require API keys set as environment variables:
 
 | Provider | Environment Variable | Required | How to Get |
 |----------|---------------------|----------|------------|
 | Semantic Scholar | `S2_API_KEY` | No | [api.semanticscholar.org](https://api.semanticscholar.org) |
-| OpenAlex | `OPENALEX_EMAIL` | No | Any email (for polite pool) |
+| OpenAlex | `OPENALEX_API_KEY`, `SCHOLAR_EMAIL` | No | [openalex.org/settings/api](https://openalex.org/settings/api) (10x daily budget); any email (polite pool) |
 | DBLP | - | No | No key needed |
+| arXiv | - | No | No key needed |
+| HAL | - | No | No key needed |
+| SSRN, bioRxiv, medRxiv, ChemRxiv, Research Square, Preprints.org, IACR | `SCHOLAR_EMAIL` | No | Any email (they search via OpenAlex) |
 | Web of Science | `WOS_EXPANDED_API_KEY` or `WOS_STARTER_API_KEY` | Yes | [developer.clarivate.com](https://developer.clarivate.com) |
 | IEEE Xplore | `IEEE_API_KEY` | Yes | [developer.ieee.org](https://developer.ieee.org) |
+| Scopus | `SCOPUS_API_KEY` | Yes | [dev.elsevier.com](https://dev.elsevier.com). Searching works from any network, but abstracts, keywords and full author lists come only with Scopus's COMPLETE view, which is bound to the institution's IP range (or `SCOPUS_INST_TOKEN`). scholar asks for it and falls back by itself (`scholar providers check` tells which view you get); `SCOPUS_VIEW=standard` or `complete` overrides. A 400 "Error translating query" on *every* query is an Elsevier-side refusal, not a query error: it has cleared by itself before, so retry later |
 
 View provider status:
 
@@ -220,6 +306,38 @@ View provider status:
 scholar providers
 ```
 
+### Network: proxy and User-Agent
+
+To reach subscriber-only content from off campus (Scopus's COMPLETE
+view, a publisher that admits the institution's IP range), route
+scholar's requests through a proxy on that network, for example an SSH
+SOCKS tunnel:
+
+```bash
+ssh -D 127.0.0.1:1081 -N you@login.your-university.example &
+scholar --proxy socks5h://127.0.0.1:1081 search "..."
+export SCHOLAR_PROXY=socks5h://127.0.0.1:1081   # or for every command
+scholar providers check --show-ip                # route, exit IP, health
+```
+
+`--proxy URL` (a global option, before the command) wins over
+`SCHOLAR_PROXY`; with neither, the standard `HTTP(S)_PROXY`/`ALL_PROXY`
+variables apply as usual. Use `socks5h://` rather than `socks5://` so
+host names are resolved by the proxy, on the far side of the tunnel.
+SOCKS needs the PySocks package (`pipx inject scholarcli PySocks`);
+scholar refuses a SOCKS proxy without it rather than failing every
+request. `scholar providers check` always prints the route; `--show-ip`
+also asks api.ipify.org for the exit address (a third-party request, so
+only on demand).
+
+Requests identify themselves as
+`scholar/<version> (+https://github.com/dbosk/scholar)`. Your
+`SCHOLAR_EMAIL` is not in it, so publishers never see the address; only
+the services that ask for one (OpenAlex, Crossref, Unpaywall) get it,
+through their own parameters. Set `SCHOLAR_USER_AGENT` to replace it. Scholar does not imitate a browser: a Cloudflare challenge
+or a login page from a publisher means the PDF is not available to
+scripts, and scholar stops there.
+
 ## Usage Examples
 
 ### Basic Search
@@ -249,6 +367,15 @@ scholar sessions resume "privacy-ml-review"
 
 # 4. Generate reports
 scholar sessions export "privacy-ml-review" -f all
+
+# 5. Synthesise the kept papers (needs a research context); the LaTeX
+#    document ends with the decision record as an appendix
+scholar llm context "privacy-ml-review" "How is privacy preserved in ML?"
+scholar llm synthesize "privacy-ml-review" -f latex -o synthesis.tex
+
+# ... or as a fragment to \input into a thesis chapter (also writes
+#     synthesis.bib; add \addbibresource{synthesis.bib} to the preamble)
+scholar llm synthesize "privacy-ml-review" -f latex --no-standalone -o synthesis.tex
 ```
 
 ### Enriching Results
@@ -263,6 +390,22 @@ scholar search "query" --enrich
 scholar enrich "session-name"
 ```
 
+When no PDF can be found for a paper, enrichment falls back to preprint
+versions: authoritative links (OpenAlex's alternate locations and
+Crossref's `has-preprint` relation) are always followed, and with
+`--preprint-search` the preprint servers are additionally searched by
+title and authors (exact match only). An adopted preprint *replaces*
+the entry — its DOI, venue, and PDF — so exported citations point at
+the version you actually read; the published DOI is kept on the entry
+for provenance, and `scholar verify` reports such entries as
+`preprint of <doi>` rather than superseded.
+
+```bash
+# Also search preprint servers for papers without a PDF
+scholar enrich "session-name" --preprint-search
+scholar search "query" --enrich --preprint-search
+```
+
 ### PDF Management
 
 ```bash
@@ -299,6 +442,12 @@ For large result sets, Scholar can use LLMs to assist with paper classification:
 # In the TUI, press 'L' to invoke LLM classification
 # Or use the CLI command directly
 scholar llm classify "session-name" --count 10
+
+# Constrain the tags to a vocabulary (tags already in the session count too)
+scholar llm classify "session-name" -t supports-claim -t qualifies-claim \
+    -t adjacent-subtopic -t off-topic-false-hit
+# ... or let the model add tags of its own when none of them fits
+scholar llm classify "session-name" --allow-new-tags
 ```
 
 ### How It Works