zotkit 0.4.2__tar.gz → 0.5.0__tar.gz

{zotkit-0.4.2/zotkit.egg-info → zotkit-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zotkit
-Version: 0.4.2
+Version: 0.5.0
 Summary: Headless Zotero library management: Web API CRUD plus direct WebDAV attachment upload/download — no desktop app required
 Author: Shawn
 License-Expression: MIT
@@ -113,6 +113,9 @@ zotkit create --file papers.json            # batch from JSON: dry-run preview
 zotkit create --file papers.json --apply    # create (dedups by DOI/title)
 zotkit attach --from papers.created.json --all   # upload the PDFs to WebDAV
 
+zotkit enrich --key AB12CD34 EF56GH78       # fill missing fields on existing items
+                                            #   in place (dry-run; --apply to write)
+
 zotkit attach --key AB12CD34 --pdf paper.pdf     # single attach
 zotkit fetch --key AB12CD34 --out downloads      # download attachment from WebDAV
 
@@ -144,7 +147,31 @@ request layer — batches use one arXiv metadata request and space PDF downloads
 per arXiv's terms of use, so callers (humans or agents) never pace themselves. A
 bad id fails alone, not the batch; the exit code is non-zero only if something
 failed. Fetching metadata from arbitrary web pages is out of scope (zotkit stays
-a daemon-free CLI — no translation-server).
+a daemon-free CLI — no translation-server). CrossRef requests identify themselves
+to the polite pool with a contact address; that should be reachable, so if you
+distribute a tool built on zotkit, set your own via `ZOTKIT_MAILTO`.
+
+### Enriching existing items
+
+`zotkit enrich --key K [K …]` completes incomplete items — missing abstracts,
+DOIs, truncated author lists — from the same arXiv/CrossRef sources, **in
+place**. The item key never changes (keys are the stable handle downstream
+tooling references; delete-and-recreate is not an option), and writes carry the
+item version, so a concurrent edit fails that item loudly instead of clobbering.
+
+The merge is deliberately conservative: only empty fields are filled; creators
+are extended only when the current list is a same-order prefix of the
+authoritative one (the classic truncated-list case — any other difference is
+reported, not touched); tags, collections, relations, and attachments are never
+modified; Extra only gains lines. Abstracts zotkit writes are stamped
+`abstract-source: …` in Extra — a stamp with no abstract means someone
+deliberately removed it, and enrich will not re-add it (reported as NEEDS
+OWNER). Items with no DOI or arXiv id are reported as needs-identifier.
+
+`--rebuild-record` additionally upgrades a preprint whose paper has since been
+published (journal DOI present) to the journal record **in the same item**:
+itemType, published title/date/venue/volume/pages — with `arXiv: <id>` appended
+to Extra and attachments untouched.
 
 Item JSON for `zotkit create --file` (a list, one object per reference):
 

{zotkit-0.4.2 → zotkit-0.5.0}/README.md

@@ -93,6 +93,9 @@ zotkit create --file papers.json            # batch from JSON: dry-run preview
 zotkit create --file papers.json --apply    # create (dedups by DOI/title)
 zotkit attach --from papers.created.json --all   # upload the PDFs to WebDAV
 
+zotkit enrich --key AB12CD34 EF56GH78       # fill missing fields on existing items
+                                            #   in place (dry-run; --apply to write)
+
 zotkit attach --key AB12CD34 --pdf paper.pdf     # single attach
 zotkit fetch --key AB12CD34 --out downloads      # download attachment from WebDAV
 
@@ -124,7 +127,31 @@ request layer — batches use one arXiv metadata request and space PDF downloads
 per arXiv's terms of use, so callers (humans or agents) never pace themselves. A
 bad id fails alone, not the batch; the exit code is non-zero only if something
 failed. Fetching metadata from arbitrary web pages is out of scope (zotkit stays
-a daemon-free CLI — no translation-server).
+a daemon-free CLI — no translation-server). CrossRef requests identify themselves
+to the polite pool with a contact address; that should be reachable, so if you
+distribute a tool built on zotkit, set your own via `ZOTKIT_MAILTO`.
+
+### Enriching existing items
+
+`zotkit enrich --key K [K …]` completes incomplete items — missing abstracts,
+DOIs, truncated author lists — from the same arXiv/CrossRef sources, **in
+place**. The item key never changes (keys are the stable handle downstream
+tooling references; delete-and-recreate is not an option), and writes carry the
+item version, so a concurrent edit fails that item loudly instead of clobbering.
+
+The merge is deliberately conservative: only empty fields are filled; creators
+are extended only when the current list is a same-order prefix of the
+authoritative one (the classic truncated-list case — any other difference is
+reported, not touched); tags, collections, relations, and attachments are never
+modified; Extra only gains lines. Abstracts zotkit writes are stamped
+`abstract-source: …` in Extra — a stamp with no abstract means someone
+deliberately removed it, and enrich will not re-add it (reported as NEEDS
+OWNER). Items with no DOI or arXiv id are reported as needs-identifier.
+
+`--rebuild-record` additionally upgrades a preprint whose paper has since been
+published (journal DOI present) to the journal record **in the same item**:
+itemType, published title/date/venue/volume/pages — with `arXiv: <id>` appended
+to Extra and attachments untouched.
 
 Item JSON for `zotkit create --file` (a list, one object per reference):
 

{zotkit-0.4.2 → zotkit-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "zotkit"
-version = "0.4.2"
+version = "0.5.0"
 description = "Headless Zotero library management: Web API CRUD plus direct WebDAV attachment upload/download — no desktop app required"
 readme = "README.md"
 license = "MIT"

{zotkit-0.4.2 → zotkit-0.5.0}/zotkit/__init__.py

@@ -2,4 +2,4 @@
 from .core import Conventions, TagConventionError, Zot, lint_tags, load_conventions
 
 __all__ = ["Zot", "lint_tags", "load_conventions", "Conventions", "TagConventionError"]
-__version__ = "0.4.2"
+__version__ = "0.5.0"

{zotkit-0.4.2 → zotkit-0.5.0}/zotkit/cli.py

@@ -1,6 +1,6 @@
 """zotkit — CLI over zotkit.core.Zot.
 
-Subcommands: doctor, find, create, attach, fetch, tag, status, move, backup, lint.
+Subcommands: doctor, find, create, enrich, attach, fetch, tag, status, move, backup, lint.
 Write commands print what they did; `create` is dry-run unless --apply.
 """
 from __future__ import annotations
@@ -97,6 +97,15 @@ def main(argv=None):
     p.add_argument("--tags", help="with --arxiv/--doi: comma-separated tags, "
                                   "e.g. field:ml,status:to-read")
 
+    p = sub.add_parser("enrich", help="fill missing fields on existing items from "
+                                      "arXiv/CrossRef, in place — the item key never "
+                                      "changes (dry unless --apply)")
+    p.add_argument("--key", nargs="+", required=True, metavar="KEY")
+    p.add_argument("--apply", action="store_true")
+    p.add_argument("--rebuild-record", action="store_true",
+                   help="also upgrade preprint items with a journal DOI to the "
+                        "journal record, in place (version of record)")
+
     p = sub.add_parser("attach", help="attach a local file to an item (WebDAV or Zotero Storage)")
     p.add_argument("--key"); p.add_argument("--pdf")
     p.add_argument("--from", dest="src", help="a .created.json list (batch)")
@@ -241,6 +250,54 @@ def main(argv=None):
             print(f"batch: created {len(new)}, skipped {dup} dup, failed {len(failures)}")
         return 1 if failures else 0
 
+    elif a.cmd == "enrich":
+        from .enrich import apply_plan, plan_enrich
+        counts = {"enriched": 0, "needs-identifier": 0, "stale-stamp": 0,
+                  "up-to-date": 0, "failed": 0}
+        for key in a.key:
+            try:
+                p_ = plan_enrich(zot, key, rebuild_record=a.rebuild_record)
+            except Exception as e:  # bad key, metadata lookup failure, …
+                counts["failed"] += 1
+                print(f"{key}  FAILED: {e}", file=sys.stderr)
+                continue
+            print(f"{key}  {p_['title'][:60]}")
+            for n in p_["notes"]:
+                print(f"    {n}")
+            if p_["status"] in ("needs-identifier", "stale-stamp", "up-to-date"):
+                if p_["status"] == "up-to-date":
+                    print("    up-to-date")
+                counts[p_["status"]] += 1
+                continue
+            if p_["rebuild"]:
+                print(f"    rebuild: {p_['item']['data']['itemType']} → "
+                      f"{p_['rebuild']} ({p_['auth'].get('DOI', '')})")
+            if p_["fills"]:
+                shown = [f + (f" (source: {p_['abstract_source']})"
+                              if f == "abstractNote" and p_["abstract_source"] else "")
+                         for f in p_["fills"]]
+                print(f"    will fill: {', '.join(shown)}")
+            for line in p_["extra_lines"]:
+                print(f"    extra += {line!r}")
+            if not a.apply:
+                continue
+            try:
+                apply_plan(zot, p_)
+                counts["enriched"] += 1
+                print("    enriched ✓")
+            except Exception as e:
+                counts["failed"] += 1
+                print(f"{key}  FAILED to write: {e}", file=sys.stderr)
+        if not a.apply:
+            print("DRY — add --apply to write.")
+        skipped = (counts["needs-identifier"] + counts["stale-stamp"]
+                   + counts["up-to-date"])
+        print(f"enriched {counts['enriched']} / skipped {skipped} "
+              f"(needs-identifier {counts['needs-identifier']}, "
+              f"stale-stamp {counts['stale-stamp']}, "
+              f"up-to-date {counts['up-to-date']}) / failed {counts['failed']}")
+        return 1 if counts["failed"] else 0
+
     elif a.cmd == "attach":
         if a.key and a.pdf:
             print(zot.attach(a.key, a.pdf))

zotkit-0.5.0/zotkit/enrich.py

@@ -0,0 +1,243 @@
+"""zotkit.enrich — complete existing items in place from arXiv/CrossRef.
+
+The core invariant: **the item key never changes**. Item keys are referenced by
+downstream manifests, so delete-and-recreate is never acceptable; every change
+here is an in-place fetch→modify→update carrying the item version (a concurrent
+edit fails that one item with 412 instead of clobbering).
+
+Merge policy:
+- only empty fields are filled — existing non-empty values are never overwritten
+  (exception: --rebuild-record, which deliberately adopts the published title,
+  date, and venue when upgrading a preprint to its version of record);
+- creators are extended only when the current list is a same-order prefix of the
+  authoritative one (the classic truncated-author-list case); any other
+  difference is reported, not touched;
+- an `abstract-source:` stamp with no abstract means an owner deliberately
+  removed the abstract — never re-add it (reported as NEEDS OWNER);
+- Extra only ever gains lines (arXiv: / abstract-source: / Repository:),
+  existing lines are never rewritten;
+- tags, collections, relations, and child items are never touched.
+
+Network requests reuse zotkit.metadata's fetchers and its per-host rate limit.
+"""
+from __future__ import annotations
+
+import re
+import time
+from typing import Any
+
+from . import metadata as md
+
+# fields the plain fill pass may write; everything else needs a dedicated rule
+_PROTECTED = {"itemType", "title", "creators", "abstractNote", "extra",
+              "tags", "collections", "relations",
+              "key", "version", "dateAdded", "dateModified"}
+
+_ARXIV_IN_TEXT = re.compile(
+    r"arxiv:\s*((?:\d{4}\.\d{4,5}|[a-z-]+(?:\.[A-Z]{2})?/\d{7})(?:v\d+)?)",
+    re.IGNORECASE)
+_STAMP = re.compile(r"^abstract-source:\s*(\w+)", re.IGNORECASE | re.MULTILINE)
+
+
+def _find_arxiv_id(d: dict) -> str | None:
+    """arXiv identity of an item: archiveID, an `arXiv:` Extra line, an
+    arxiv.org url, or the id embedded in arXiv's own DataCite DOI."""
+    for text in (d.get("archiveID", ""), d.get("extra", "")):
+        m = _ARXIV_IN_TEXT.search(text)
+        if m:
+            return m.group(1)
+    if "arxiv.org/" in d.get("url", ""):
+        try:
+            return md.arxiv_id(d["url"])
+        except md.MetadataError:
+            pass
+    doi = (d.get("DOI") or "").strip()
+    if doi.lower().startswith("10.48550/arxiv."):
+        return doi[len("10.48550/arxiv."):]
+    return None
+
+
+def _norm(s: str | None) -> str:
+    return re.sub(r"\s+", " ", s or "").strip().casefold()
+
+
+def _same_creator(c: dict, a: dict) -> bool:
+    """Loose identity: equal last name plus compatible first name (initials
+    match full names), so 'A. Einstein' matches 'Albert Einstein'."""
+    if c.get("creatorType") != a.get("creatorType"):
+        return False
+    if c.get("name") or a.get("name"):
+        full = lambda x: _norm(x.get("name")
+                               or f"{x.get('firstName', '')} {x.get('lastName', '')}")
+        return full(c) == full(a)
+    if _norm(c.get("lastName")) != _norm(a.get("lastName")):
+        return False
+    cf, af = _norm(c.get("firstName")), _norm(a.get("firstName"))
+    return not cf or not af or cf[0] == af[0]
+
+
+def _real_creators(creators: list[dict]) -> list[dict]:
+    """Drop all-empty placeholder creators (an untouched item template ships
+    one empty author)."""
+    return [c for c in creators
+            if any(v for k, v in c.items() if k != "creatorType")]
+
+
+def _fetch_authoritative(d: dict) -> tuple[dict | None, str | None, str | None]:
+    """→ (authoritative record, arXiv id, skip reason). Journal DOI wins
+    (version of record); arXiv otherwise. The arXiv fetch itself may upgrade
+    to a journal record per v0.4.2 rules — equally welcome here."""
+    doi = (d.get("DOI") or "").strip()
+    aid = _find_arxiv_id(d)
+    journal_doi = doi if doi and not doi.lower().startswith("10.48550/") else None
+    if journal_doi:
+        auth = md.fetch_doi(journal_doi)
+        # CrossRef abstracts are often absent; borrow arXiv's when we can
+        if not auth.get("abstractNote") and aid and not d.get("abstractNote"):
+            r = md.fetch_arxiv_batch([aid])[0]
+            pre = r.get("item") or {}
+            if pre.get("abstractNote"):
+                auth["abstractNote"] = pre["abstractNote"]
+                md._add_extra(auth, "abstract-source: arxiv")
+        return auth, aid, None
+    if aid:
+        r = md.fetch_arxiv_batch([aid])[0]
+        if r.get("error"):
+            raise md.MetadataError(r["error"])
+        return r["item"], aid, None
+    return None, None, "no DOI or arXiv id on the item"
+
+
+def plan_enrich(zot, key: str, *, rebuild_record: bool = False) -> dict[str, Any]:
+    """Compute what enriching `key` would change. Returns a plan dict:
+    status 'plan' (something to do), 'up-to-date', 'needs-identifier',
+    'stale-stamp', with fills / creators / extra_lines / rebuild / notes.
+    Raises MetadataError (lookup) or pyzotero errors (bad key) — callers
+    turn those into per-item failures."""
+    item = zot.z.item(key)
+    d = item["data"]
+    p: dict[str, Any] = {"key": key, "title": d.get("title", ""), "item": item,
+                         "fills": {}, "extra_lines": [], "creators": None,
+                         "rebuild": None, "notes": [], "abstract_source": None}
+
+    auth, aid, skip = _fetch_authoritative(d)
+    if auth is None:
+        p["status"] = "needs-identifier"
+        p["notes"].append(f"SKIP needs-identifier: {skip}")
+        return p
+    p["auth"], p["aid"] = auth, aid
+
+    # ---- abstract (stamp rules) ----
+    stamped = _STAMP.search(d.get("extra", ""))
+    auth_ab_src = (_STAMP.search(auth.get("extra", "")) or [None, None])[1]
+    if not d.get("abstractNote"):
+        if stamped:
+            p["notes"].append("NEEDS OWNER stale-stamp: abstract-source stamp "
+                              "present but abstract empty — not re-adding")
+            p["stale_stamp"] = True
+        elif auth.get("abstractNote"):
+            p["fills"]["abstractNote"] = auth["abstractNote"]
+            p["abstract_source"] = (auth_ab_src or "").lower() or None
+            if p["abstract_source"]:
+                p["extra_lines"].append(f"abstract-source: {p['abstract_source']}")
+
+    # ---- plain fields: fill only what's empty ----
+    for f, v in auth.items():
+        if f in _PROTECTED or not v:
+            continue
+        if f in d and not d.get(f):
+            p["fills"][f] = v
+    if not d.get("title") and auth.get("title"):
+        p["fills"]["title"] = auth["title"]
+
+    # ---- creators: whole-fill or prefix extension only ----
+    cur = _real_creators(d.get("creators", []))
+    authc = auth.get("creators", [])
+    if authc:
+        if not cur:
+            p["creators"] = authc
+        elif (len(cur) < len(authc)
+              and all(_same_creator(c, a) for c, a in zip(cur, authc))):
+            p["creators"] = authc
+            p["notes"].append(f"creators: {len(cur)} → {len(authc)} "
+                              "(authoritative prefix extension)")
+        elif not (len(cur) == len(authc)
+                  and all(_same_creator(c, a) for c, a in zip(cur, authc))):
+            p["notes"].append(f"creators differ from authoritative record "
+                              f"({len(cur)} vs {len(authc)}) — left untouched")
+
+    # ---- version-of-record rebuild (explicit opt-in) ----
+    if (rebuild_record and d.get("itemType") == "preprint"
+            and auth.get("itemType") not in (None, "preprint")):
+        p["rebuild"] = auth["itemType"]
+
+    # ---- arXiv identity line ----
+    if aid:
+        # a rebuild drops archiveID (no such field on journal types), so only
+        # Extra itself counts as "already recorded" in that case
+        known = d.get("extra", "") + ("" if p["rebuild"]
+                                      else " " + d.get("archiveID", ""))
+        if f"arxiv:{aid.lower()}" not in known.lower().replace(" ", ""):
+            p["extra_lines"].append(f"arXiv: {aid}")
+
+    p["status"] = ("plan" if (p["fills"] or p["creators"] is not None
+                              or p["extra_lines"] or p["rebuild"])
+                   else "stale-stamp" if p.get("stale_stamp")
+                   else "up-to-date")
+    return p
+
+
+def apply_plan(zot, p: dict[str, Any]) -> None:
+    """Execute a 'plan' plan in place. The item key never changes; the write
+    carries the fetched version, so a concurrent edit fails loudly (412)."""
+    item, d, auth = p["item"], p["item"]["data"], p.get("auth", {})
+    children_before = len(zot.z.children(p["key"])) if p["rebuild"] else None
+
+    if p["rebuild"]:
+        tmpl = zot.z.item_template(p["rebuild"])
+        old = dict(d)
+        keep = {k: old[k] for k in ("key", "version", "dateAdded", "dateModified")
+                if k in old}
+        new: dict[str, Any] = {}
+        for f in tmpl:
+            if f in ("tags", "collections", "relations"):
+                new[f] = old.get(f, tmpl[f])       # never touched
+            elif f == "extra":
+                new[f] = old.get("extra", "")
+            elif f == "creators":
+                new[f] = (p["creators"] if p["creators"] is not None
+                          else old.get("creators", []))
+            elif f == "abstractNote":  # stale stamp forbids re-adding here too
+                new[f] = old.get("abstractNote") or (
+                    "" if p.get("stale_stamp") else auth.get("abstractNote", ""))
+            else:
+                # version of record wins: published title/date/venue over preprint's
+                new[f] = auth.get(f) or old.get(f) or tmpl[f]
+        if old.get("repository") and not p.get("aid"):
+            md._add_extra(new, f"Repository: {old['repository']}")
+        d.clear()
+        d.update(keep)
+        d.update(new)
+    else:
+        d.update(p["fills"])
+        if p["creators"] is not None:
+            d["creators"] = p["creators"]
+
+    for line in p["extra_lines"]:
+        md._add_extra(d, line)
+
+    zot.z.update_item(item)
+
+    if children_before is not None:
+        # the API can serve a stale (empty) children list right after the PUT —
+        # tolerate brief read-after-write lag before declaring a real mismatch
+        for attempt in range(4):
+            children_after = len(zot.z.children(p["key"]))
+            if children_after == children_before:
+                break
+            time.sleep(2)
+        if children_after != children_before:
+            raise RuntimeError(
+                f"child-item count changed during rebuild of {p['key']} "
+                f"({children_before} → {children_after}) — investigate before "
+                "trusting this item")

{zotkit-0.4.2 → zotkit-0.5.0}/zotkit/metadata.py

@@ -14,6 +14,7 @@ batch loops — never need to pace themselves.
 """
 from __future__ import annotations
 
+import os
 import re
 import time
 import xml.etree.ElementTree as ET
@@ -23,7 +24,9 @@ import httpx
 
 from . import __version__
 
-CONTACT = "mailto:Eric.M.990909@gmail.com"
+# CrossRef's polite pool wants a reachable contact; $ZOTKIT_MAILTO overrides
+# the default (distributors of zotkit-based tools: please set your own).
+CONTACT = f"mailto:{os.environ.get('ZOTKIT_MAILTO', 'ccmuuc@gmail.com')}"
 USER_AGENT = f"zotkit/{__version__} (https://github.com/oldantique/zotkit; {CONTACT})"
 
 _ATOM = "{http://www.w3.org/2005/Atom}"

{zotkit-0.4.2 → zotkit-0.5.0/zotkit.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zotkit
-Version: 0.4.2
+Version: 0.5.0
 Summary: Headless Zotero library management: Web API CRUD plus direct WebDAV attachment upload/download — no desktop app required
 Author: Shawn
 License-Expression: MIT
@@ -113,6 +113,9 @@ zotkit create --file papers.json            # batch from JSON: dry-run preview
 zotkit create --file papers.json --apply    # create (dedups by DOI/title)
 zotkit attach --from papers.created.json --all   # upload the PDFs to WebDAV
 
+zotkit enrich --key AB12CD34 EF56GH78       # fill missing fields on existing items
+                                            #   in place (dry-run; --apply to write)
+
 zotkit attach --key AB12CD34 --pdf paper.pdf     # single attach
 zotkit fetch --key AB12CD34 --out downloads      # download attachment from WebDAV
 
@@ -144,7 +147,31 @@ request layer — batches use one arXiv metadata request and space PDF downloads
 per arXiv's terms of use, so callers (humans or agents) never pace themselves. A
 bad id fails alone, not the batch; the exit code is non-zero only if something
 failed. Fetching metadata from arbitrary web pages is out of scope (zotkit stays
-a daemon-free CLI — no translation-server).
+a daemon-free CLI — no translation-server). CrossRef requests identify themselves
+to the polite pool with a contact address; that should be reachable, so if you
+distribute a tool built on zotkit, set your own via `ZOTKIT_MAILTO`.
+
+### Enriching existing items
+
+`zotkit enrich --key K [K …]` completes incomplete items — missing abstracts,
+DOIs, truncated author lists — from the same arXiv/CrossRef sources, **in
+place**. The item key never changes (keys are the stable handle downstream
+tooling references; delete-and-recreate is not an option), and writes carry the
+item version, so a concurrent edit fails that item loudly instead of clobbering.
+
+The merge is deliberately conservative: only empty fields are filled; creators
+are extended only when the current list is a same-order prefix of the
+authoritative one (the classic truncated-list case — any other difference is
+reported, not touched); tags, collections, relations, and attachments are never
+modified; Extra only gains lines. Abstracts zotkit writes are stamped
+`abstract-source: …` in Extra — a stamp with no abstract means someone
+deliberately removed it, and enrich will not re-add it (reported as NEEDS
+OWNER). Items with no DOI or arXiv id are reported as needs-identifier.
+
+`--rebuild-record` additionally upgrades a preprint whose paper has since been
+published (journal DOI present) to the journal record **in the same item**:
+itemType, published title/date/venue/volume/pages — with `arXiv: <id>` appended
+to Extra and attachments untouched.
 
 Item JSON for `zotkit create --file` (a list, one object per reference):
 

{zotkit-0.4.2 → zotkit-0.5.0}/zotkit.egg-info/SOURCES.txt

@@ -4,6 +4,7 @@ pyproject.toml
 zotkit/__init__.py
 zotkit/cli.py
 zotkit/core.py
+zotkit/enrich.py
 zotkit/metadata.py
 zotkit.egg-info/PKG-INFO
 zotkit.egg-info/SOURCES.txt