linkml-reference-validator 0.2.1rc2__tar.gz → 0.2.1rc3__tar.gz

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.2.1rc3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: linkml_reference_validator
-Version: 0.2.1rc2
+Version: 0.2.1rc3
 Summary: Validation of supporting text from references and publications
 Author-email: Chris Mungall <cjmungall@lbl.gov>
 License-Expression: Apache-2.0

linkml_reference_validator-0.2.1rc3/docs/how-to/export-cache-to-zotero.md

@@ -0,0 +1,114 @@
+# Export a Reference Cache to Zotero
+
+This workflow turns a checked-in public reference cache into a Zotero collection,
+asks Zotero to find available PDFs, and then inventories the resulting attachments
+without putting closed manuscripts into the project repository.
+
+The three stores remain separate:
+
+| Store | Purpose | Safe to commit? |
+|---|---|---:|
+| Project `references_cache` | Reproducible public validation evidence | Yes |
+| Zotero library | User-managed references and PDF attachments | No |
+| Private research cache | Extracted user-library text for agent context | No |
+
+Normal validation reads only the first store.
+
+## 1. Export metadata from the public cache
+
+Run the exporter from the `linkml-reference-validator` checkout. For dismech:
+
+```bash
+uv run linkml-reference-validator cache export \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --needs-full-text \
+  --output /Users/cjm/Downloads/dismech-zotero.json
+```
+
+`--needs-full-text` is the default. It excludes cache entries that already have
+full text, excludes non-publication identifiers, and deduplicates by normalized
+DOI and then PMID. The CSL JSON output uses an explicit metadata allowlist:
+
+- title and authors;
+- journal and year;
+- DOI; or
+- PMID and its public PubMed URL.
+
+It never contains cached article text, excerpts, provenance, PDFs, local paths,
+or private-cache data. The command refuses to replace an existing output file;
+use `--force` only after checking the destination. Use `--all` if you explicitly
+want eligible references that already contain full text.
+
+## 2. Import the file into Zotero
+
+In Zotero:
+
+1. Choose **File → Import**.
+2. Choose **A file**.
+3. Select `dismech-zotero.json`.
+4. Keep the imported records in a project-specific collection such as
+   **dismech**.
+
+CSL JSON is a standard format supported by Zotero. Importing the file adds only
+bibliographic records; it does not copy anything from the validator's caches.
+
+## 3. Ask Zotero to find PDFs
+
+Select one or more top-level journal articles in the center item list,
+right-click the selection, and choose **Find Full Text**. It is a built-in Zotero
+command, not a plugin.
+
+![Zotero Find Full Text processing a batch of imported references](../assets/zotero-find-full-text.png)
+
+The command appears for an eligible parent item that has a DOI and does not
+already have a file attachment. It may also be unavailable in a group library
+where files cannot be added. If it is absent, first test a single article in
+your personal library and confirm that its DOI field is populated.
+
+For a large import, start with a modest batch rather than all records at once.
+Publishers can temporarily rate-limit repeated requests, and a later retry may
+find additional files. Zotero reports **Full Text PDF** or **No file found** for
+each attempted item.
+
+## 4. Inventory the enriched Zotero library
+
+Once Zotero finishes, scan the project cache again:
+
+```bash
+uv run linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --dry-run
+```
+
+This read-only scan matches exact DOI, PMID, or PMCID identifiers and reports
+which cached references now have usable Zotero text or PDF attachments. It does
+not invoke **Find Full Text** and does not modify Zotero.
+
+After reviewing the matches, materialize them into the separate private research
+cache:
+
+```bash
+uv run linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir /Users/cjm/repos/dismech/references_cache \
+  --apply
+```
+
+Apply mode leaves `references_cache` unchanged. Private content goes to
+`~/.cache/linkml-reference-validator/private` by default with owner-only
+permissions. Agents may inspect it for context, but validation cannot use it as
+evidence.
+
+## 5. Validate reproducibly
+
+Run validation normally. Results remain reproducible because the validator
+reads the public project cache and accepts only public full-text provider results.
+Neither the contents of Zotero nor the machine-specific private cache affect
+validation.
+
+## See also
+
+- [Fetching Full Text and PDFs](fetch-full-text-and-pdfs.md)
+- [Zotero: importing standardized formats](https://www.zotero.org/support/kb/importing_standardized_formats)
+- [Zotero: adding files to a library](https://www.zotero.org/support/attaching_files)

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.2.1rc3}/docs/how-to/fetch-full-text-and-pdfs.md

@@ -46,9 +46,11 @@ your config YAML or on the `ReferenceValidationConfig` object):
 |-----|---------|-------------|
 | `fetch_full_text` | `true` | Attempt to obtain full text via the provider chain when a metadata source does not already return full text. |
 | `full_text_providers` | `[pmc, epmc_preprint, unpaywall, openalex]` | Ordered list of provider names to try until one yields usable full text. |
+| `private_cache_dir` | `~/.cache/linkml-reference-validator/private` | Separate research cache for closed/user-library full text; never read by validation. |
 | `pdf_backend` | `pypdf` | Name of the PDF text-extraction backend. |
 | `download_pdfs` | `true` | If true, persist downloaded PDFs to the files cache directory. |
 | `full_text_providers_file` | `null` | Optional path to a YAML file defining custom full-text providers. |
+| `zotero_base_url` | `http://localhost:23119/api/users/0` | Read-only Zotero local API library used by the opt-in `zotero` provider. |
 
 Two existing keys are also reused by the full-text machinery:
 
@@ -71,6 +73,106 @@ pdf_backend: pypdf
 download_pdfs: true
 ```
 
+## Private manuscripts in Zotero
+
+The opt-in **`zotero`** provider can find non-open manuscripts that are already
+in your personal Zotero library. It matches exact DOI, PMID, or PMCID identifiers;
+it does not automatically accept fuzzy title matches.
+
+Zotero must be running, and its local API must be enabled in Zotero's advanced
+settings. Inventory the reference cache before changing it:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --dry-run
+```
+
+The report distinguishes `found`, `not_found`, `already_full_text`, and `error`.
+Dry-run is the default and never changes cache files. After reviewing exact
+matches, apply usable text with:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --apply
+```
+
+Apply mode does **not** modify `references_cache`. It writes the enriched entry
+to a separate research cache at `~/.cache/linkml-reference-validator/private`
+by default. Agents may inspect that cache for background context, but ordinary
+validation never reads it and never cites it as evidence. Validation therefore
+has the same inputs locally, in CI, and on another contributor's machine.
+
+To keep the research cache in a private repository or another secured location:
+
+```bash
+linkml-reference-validator cache enrich \
+  --provider zotero \
+  --cache-dir references_cache \
+  --private-cache-dir /path/to/private-reference-cache \
+  --apply
+```
+
+The same location can be configured for all commands:
+
+```yaml
+cache_dir: references_cache
+private_cache_dir: /path/to/private-reference-cache
+```
+
+Zotero-indexed text is preferred. If Zotero has not indexed an attachment, the
+provider downloads it from Zotero's local PDF endpoint and sends it through the
+normal PDF extractor. The source attachment key and `user_library` access type
+are persisted, but the ephemeral localhost URL is not.
+
+!!! warning "Keep research caches private"
+
+    Private cache entries contain extracted manuscript text and may contain a
+    copied PDF. Do not place this cache in a public repository or share it
+    unless you have permission to redistribute that content.
+
+Do not add `zotero` to `full_text_providers`. Even if it is configured there,
+ordinary validation rejects its `user_library` result and continues to public
+providers. Use `cache enrich` when you want to search Zotero.
+
+Support for reproducible validation backed by private manuscripts is a future,
+separate feature. The proposed design is a trusted offline job that emits a
+minimal signed excerpt attestation into the public cache. The validator would
+verify the signature using a checked-in public key; neither agents nor CI would
+have the signing key or permission to generate those files directly.
+
+## Export a cache to a Zotero collection
+
+Export references that still need full text as metadata-only CSL JSON:
+
+```bash
+linkml-reference-validator cache export \
+  --cache-dir references_cache \
+  --format csl-json \
+  --needs-full-text \
+  --output project-zotero.json
+```
+
+The default `--needs-full-text` mode excludes cache entries that already contain
+full text. It also excludes non-publication identifiers, deduplicates by
+normalized DOI and then PMID, and exports only an explicit metadata allowlist:
+title, authors, journal, year, DOI, PMID, and PubMed URL. Cached article text,
+excerpts, provenance, PDFs, and local paths are never exported. Use `--all` only
+when you deliberately want eligible records that already have full text.
+
+The exporter refuses to replace an existing output file unless `--force` is
+given. Review the reported counts, then in Zotero choose **File → Import → A
+file**, select the JSON file, and place the imported items in a project-specific
+collection. Zotero supports CSL JSON as a standard import format. After import,
+select the collection and run **Find Full Text**, then rerun `cache enrich`.
+
+See [Zotero's standardized-format import instructions](https://www.zotero.org/support/kb/importing_standardized_formats).
+For the complete workflow with screenshots, see
+[Export a Reference Cache to Zotero](export-cache-to-zotero.md).
+
 ## Preprints
 
 Preprints are first-class references. They are increasingly cited for early

{linkml_reference_validator-0.2.1rc2 → linkml_reference_validator-0.2.1rc3}/docs/reference/cli.md

@@ -761,6 +761,69 @@ linkml-reference-validator cache lookup PMID:16888623 --content
 linkml-reference-validator cache lookup PMID:16888623 --no-cache
 ```
 
+## cache export
+
+Export public bibliographic metadata to CSL JSON for Zotero import.
+
+```bash
+linkml-reference-validator cache export --output project-zotero.json [OPTIONS]
+```
+
+**Options:**
+
+- `--output PATH, -o PATH` - Required destination file
+- `--format TEXT` - Export format (default and currently supported: `csl-json`)
+- `--needs-full-text` - Export only publication records needing full text (default)
+- `--all` - Include eligible records that already contain full text
+- `--cache-dir PATH, -c PATH` - Public reference cache directory
+- `--force, -f` - Replace an existing output file
+- `--config PATH` - Validation configuration file
+- `--verbose, -v` - Enable detailed logging
+
+```bash
+linkml-reference-validator cache export \
+  --cache-dir references_cache \
+  --needs-full-text \
+  --output project-zotero.json
+```
+
+The output is a DOI/PMID-deduplicated metadata allowlist. It never includes
+cached article content, excerpts, PDFs, local paths, or private-cache data.
+
+---
+
+## cache enrich
+
+Inventory or enrich existing abstract-only cache entries through one full-text
+provider. This is primarily intended for opt-in private-library providers such
+as Zotero.
+
+```bash
+linkml-reference-validator cache enrich [OPTIONS]
+```
+
+**Options:**
+
+- `--provider TEXT` - Registered provider name (default: `zotero`)
+- `--cache-dir PATH, -c PATH` - Reference cache directory
+- `--private-cache-dir PATH` - Separate private research-cache destination (default: `~/.cache/linkml-reference-validator/private`)
+- `--dry-run` - Report matches without changing files (default)
+- `--apply` - Materialize usable matches into the private research cache
+- `--config PATH` - Validation configuration file
+- `--verbose, -v` - Enable detailed logging
+
+```bash
+# Safe inventory
+linkml-reference-validator cache enrich --provider zotero --dry-run
+
+# Apply reviewed exact matches
+linkml-reference-validator cache enrich --provider zotero --apply
+```
+
+The public source cache is never modified by this command, and validation never
+reads the private destination. Private-library content may be copyrighted; keep
+the research cache private.
+
 ---
 
 ## Reference ID Formats

linkml_reference_validator-0.2.1rc3/docs/superpowers/plans/2026-08-05-private-library-full-text.md

@@ -0,0 +1,190 @@
+# Private Library Full-Text Integration — Implementation Plan
+
+**Date:** 2026-08-05
+**Design:** [Private Library Full-Text Integration](../specs/2026-08-05-private-library-full-text-design.md)
+
+All implementation tasks follow test-driven development: add a failing test,
+implement the smallest production change, then run the focused test and the full
+`just test` suite. Network tests use realistic recorded/constructed API responses;
+live tests remain opt-in.
+
+## Phase 1: Zotero local API minimum viable workflow
+
+### 1. Generalize full-text locations
+
+Add tests for materializing a local PDF path, rejecting a path over the size cap,
+rejecting a non-PDF file with a `.pdf` suffix, and passing ephemeral request
+headers without persisting them.
+
+Then:
+
+- add `local_path`, `headers`, `access_type`, and `source_item_id` to
+  `FullTextLocation`;
+- teach `ReferenceFetcher._materialize()` to read a local file under the same size
+  and format-sniffing rules as HTTP content;
+- teach `ContentAcquirer` to accept per-location headers;
+- add private provenance fields to `ReferenceContent` and cache round-tripping;
+- ensure secrets and localhost URLs are not written to frontmatter.
+
+### 2. Add a Zotero API client and exact identifier index
+
+Add contract tests using representative Zotero v3 JSON for:
+
+- local API availability detection;
+- paginated top-level item loading;
+- DOI normalization;
+- PMID/PMCID extraction from structured fields and `extra`;
+- incremental index refresh metadata;
+- duplicate and ambiguous identifiers;
+- child attachment selection;
+- indexed-full-text success and PDF fallback.
+
+Then implement a small read-only client rather than coupling HTTP and matching
+logic directly inside the provider. The default base URL is
+`http://localhost:23119/api`; web mode remains disabled in this phase.
+
+### 3. Add `ZoteroFullTextProvider`
+
+Add provider tests showing:
+
+- no identifier returns `None`;
+- an exact DOI/PMID hit returns Zotero indexed text when usable;
+- short or absent indexed text returns the PDF file location;
+- ambiguity returns no location and a diagnostic;
+- connection errors remain retryable in the provider chain;
+- provenance is `provider=zotero` and `access_type=user_library` with no OA claim.
+
+Register `zotero` for the explicit `cache enrich` workflow. Ordinary validation
+must reject its `user_library` locations even if it is accidentally included in
+the configured provider chain.
+
+### 4. Add `cache enrich`
+
+Add CLI tests first for dry-run, apply, misses, ambiguity, provider-unavailable,
+and stable tabular output. Implement:
+
+```text
+cache enrich --provider zotero --cache-dir PATH [--dry-run]
+```
+
+The command enumerates Markdown cache files, loads them through `ReferenceFetcher`,
+and runs only the selected private provider even if `full_text_attempted` is
+already true. Apply mode reuses the normal materialization path but writes an
+owner-only private research cache outside the project by default.
+
+### 5. Positive local smoke test
+
+Run the dry-run command against a temporary cache entry whose DOI is known to
+exist in the local Zotero library while Zotero is running. Verify:
+
+- exact DOI match;
+- at least 500 extracted characters;
+- cache provenance and checksum;
+- no credential or private URL leakage;
+- the source Zotero library remains unchanged.
+
+The checked-in test suite must not depend on the developer's personal library.
+
+## Phase 2: Zotero Web API
+
+### 6. Add authenticated read-only web mode
+
+Test and implement user/group library configuration, pagination, `since`-based
+index refresh, backoff/rate-limit headers, attachment download authentication,
+and Zotero storage/WebDAV absence behavior.
+
+Credentials come from `ZOTERO_API_KEY` or a credential callback. Add explicit
+configuration for user/group library ID; never accept API keys in URLs.
+
+### 7. Operational documentation
+
+Document how to enable the local API, create a read-only Zotero key, scan before
+applying, protect private caches, and keep private content out of validation.
+
+## Phase 3: Paperpile local sync
+
+### 8. Build a generic local library index
+
+Extract identifier normalization and match reporting from the Zotero client into
+a source-neutral index. Add BibTeX ingestion with tests for DOI, PMID, title,
+year, author, duplicate keys, and malformed entries. Add a cacheable PDF probe
+that reads metadata/first-page text only once per file and stores its checksum.
+
+### 9. Add `PaperpileFullTextProvider` for local files
+
+Test and implement configuration for:
+
+- a local Paperpile/Google Drive PDF root;
+- a Paperpile automatic BibTeX export file or HTTPS download link;
+- an optional manual Paperpile JSON export.
+
+Only exact DOI/PMID matches apply automatically. Title/year/author candidates are
+reported by `cache enrich --dry-run` and require explicit user confirmation in a
+future task.
+
+### 10. Measure matching quality
+
+Run dry-run evaluation on a representative private cache and record precision,
+coverage, ambiguity, and unmatched reasons separately. Do not weaken matching
+thresholds to improve coverage. Use findings to decide whether guarded metadata
+matching should ever be enabled for apply mode.
+
+## Phase 4: Paperpile via Google Drive
+
+### 11. Add a read-only Drive adapter
+
+Test with Google Drive API fixtures, then implement OAuth read-only listing and
+download, incremental change tracking, native Google shortcut handling, and
+streamed size limits. Reuse the Paperpile BibTeX identity index from Phase 3.
+
+This phase must not mutate Drive or Paperpile. Tokens use the system credential
+store and are excluded from logs and config serialization.
+
+## Phase 5: Privacy hardening and signed attestations
+
+### 12. Separate private content from shareable cache metadata
+
+Completed in the first implementation: private content uses an owner-only
+research cache outside the project by default. Validation reads only the public
+cache, and its ordinary provider chain rejects explicitly non-open locations.
+
+### 13. Specify and implement offline signed excerpt attestations
+
+Define a canonical, versioned `excerpts.yaml` schema and bind each record to its
+reference ID, claim hash, normalized excerpt hash, private source checksum,
+location, result, generator version, timestamp, and key ID. A trusted offline
+job reads the private cache and signs with Ed25519. Only the verification key is
+checked in; agents and CI cannot write trusted attestations. Add tests for valid,
+tampered, wrong-reference, wrong-claim, unknown-key, expired-key, and unsigned
+artifacts before enabling this optional validation mode.
+
+## Phase 6: Browse a cache in Zotero
+
+### 14. Export cache metadata for Zotero import
+
+Implemented `cache export --format csl-json` with deterministic DOI/PMID
+deduplication and missing-full-text filtering. It exports an explicit
+bibliographic metadata allowlist and always excludes cached text, excerpts,
+provenance, PDFs, private-cache data, and local paths. The command refuses to
+overwrite an output file without `--force`.
+
+### 15. Consider an opt-in Zotero collection sync
+
+Only after export is stable, design an idempotent write integration that creates
+or updates a dedicated Zotero collection through the documented write API. It
+must require explicit write authorization and a dry-run diff; it must never
+alter unrelated Zotero items or attach private files implicitly.
+
+## Definition of done for the minimal first task
+
+- `cache enrich --provider zotero --dry-run` inventories an existing reference
+  cache without modifying it.
+- Exact DOI/PMID matches identify PDF attachments in the supported Zotero local
+  API.
+- Apply mode extracts usable text into the private research cache without changing the
+  public source cache.
+- Normal validation ignores the private research cache and rejects private
+  provider results.
+- Misses and ambiguities are visible and safe.
+- No private API, direct SQLite dependency, secret persistence, or library write.
+- Focused tests, doctests, mypy, Ruff, and the full `just test` suite pass.