zotkit 0.2.0__tar.gz → 0.4.0__tar.gz

{zotkit-0.2.0/zotkit.egg-info → zotkit-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zotkit
-Version: 0.2.0
+Version: 0.4.0
 Summary: Headless Zotero library management: Web API CRUD plus direct WebDAV attachment upload/download — no desktop app required
 Author: Shawn
 License-Expression: MIT
@@ -20,16 +20,22 @@ Dynamic: license-file
 
 # zotkit
 
+[![PyPI](https://img.shields.io/pypi/v/zotkit)](https://pypi.org/project/zotkit/)
+[![Python](https://img.shields.io/pypi/pyversions/zotkit)](https://pypi.org/project/zotkit/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
 **Headless Zotero library management — no desktop app required.**
 
+**English** | [简体中文](README.zh-CN.md)
+
 "Headless" simply means zotkit never needs the Zotero app (or any window) open: it is a
 Python library + CLI that talks straight to the
 [Zotero Web API](https://www.zotero.org/support/dev/web_api/v3/start), so you can
 search, create, tag, and organize items from any terminal — macOS, Windows, or Linux,
-your laptop or a remote server. Uniquely, if your attachments sync to a **personal
-WebDAV server**, zotkit can **upload and download the files themselves** by speaking
-Zotero's WebDAV storage format directly — the one capability the Web API does not
-provide, and (as far as we know) not offered by any other headless tool.
+your laptop or a remote server. If your attachments sync to a **personal WebDAV
+server**, zotkit can **upload and download the files themselves** by speaking
+Zotero's WebDAV storage format directly — a capability the Web API itself does not
+provide. The format is documented in [docs/webdav-format.md](docs/webdav-format.md).
 
 Built for servers, scripts, and **LLM agents**: every write is dry-run by default,
 batched, and version-checked, and you can define a tag taxonomy that is *enforced in
@@ -45,6 +51,18 @@ code* so an agent (or a tired human) can't pollute your library with inconsisten
 | Attachment files on **WebDAV** | ✅ | ⚠️ download at best | ✅ **upload + download** |
 | Tag conventions enforced in code | ❌ | ❌ | ✅ optional `conventions.toml` |
 
+## The zotkit family
+
+Two independent, complementary projects — use either on its own, or both together:
+
+| | What it is |
+|---|---|
+| **zotkit** (this repo) | Headless Python CLI + library. Runs anywhere, needs no Zotero app. |
+| **[zotkit-reader](https://github.com/oldantique/zotkit-reader)** | A Zotero Reader sidebar that embeds a Codex/Claude agent beside the PDF you're reading. Maintained by [@ChanceSiyuan](https://github.com/ChanceSiyuan). |
+
+They share a name and a philosophy, not a codebase: zotkit-reader can call zotkit over MCP
+for library-wide operations, but neither requires the other to be installed.
+
 ## Install
 
 Pure Python (3.11+), no platform-specific bits — the same package works on macOS,
@@ -70,6 +88,9 @@ Copy [`.env.example`](.env.example) to `./.env`, `~/.config/zotkit/env`, or any
   `attach`/`fetch` automatically use Zotero Storage through the Web API's upload/download
   endpoints instead. The storage mode is detected from your `.env`, nothing to configure.
 
+After filling it in, run **`zotkit doctor`** — it validates the config file, API
+access, and attachment storage, and tells you exactly what to fix if anything fails.
+
 Optionally, copy [`conventions.example.toml`](conventions.example.toml) to
 `conventions.toml` next to your `.env` to define a namespaced tag taxonomy
 (`field:physics`, `status:to-read`, …). With it in place, `zotkit create` / `zotkit tag`
@@ -81,7 +102,12 @@ Optionally, copy [`conventions.example.toml`](conventions.example.toml) to
 zotkit find --title "boson sampling"        # search by title/tag/collection
 zotkit find --tag status:to-read
 
-zotkit create --file papers.json            # dry-run: shows what would be created
+zotkit create --arxiv 2401.12345            # fetch arXiv metadata, dry-run preview
+zotkit create --arxiv 2401.12345 --apply --tags field:ai   # create + download & attach the PDF
+zotkit create --doi 10.1038/nature14539     # fetch CrossRef metadata by DOI (no PDF —
+                                            #   usually paywalled; attach manually after)
+
+zotkit create --file papers.json            # batch from JSON: dry-run preview
 zotkit create --file papers.json --apply    # create (dedups by DOI/title)
 zotkit attach --from papers.created.json --all   # upload the PDFs to WebDAV
 
@@ -96,7 +122,14 @@ zotkit backup                               # full JSON snapshot -> backups/
 zotkit lint field:physics topic:new-idea    # offline tag check
 ```
 
-Item JSON for `zotkit create` (a list, one object per reference):
+`--arxiv` takes an id or an abs/pdf URL and maps the full record (all authors,
+abstract, date, DOI, `preprint` item type); `--doi` maps CrossRef records
+(journal articles, conference papers, books, chapters, …) and refuses to guess
+on CrossRef types it doesn't know. Both accept `--collection` and `--tags`, and
+`--no-pdf` skips the arXiv PDF. Fetching metadata from arbitrary web pages is
+out of scope (zotkit stays a daemon-free CLI — no translation-server).
+
+Item JSON for `zotkit create --file` (a list, one object per reference):
 
 ```json
 [{"itemType": "journalArticle", "title": "…",
@@ -160,6 +193,16 @@ Details in [`docs/webdav-format.md`](docs/webdav-format.md).
 The format was determined by interoperability inspection of the author's own library.
 This project is not affiliated with or endorsed by Zotero.
 
+## Limits & roadmap
+
+- `find` currently lists the library client-side — instant for hundreds of items,
+  sluggish for many thousands. Server-side search is planned.
+- Group libraries should work for item operations (untested); WebDAV file sync is
+  personal-libraries-only (a Zotero limitation).
+- `--doi`/`--arxiv` import covers arXiv + CrossRef; DataCite-only DOIs and
+  arbitrary-URL scraping (translation-server territory) are out of scope.
+- Planned: an MCP server wrapper, server-side search.
+
 ## License
 
 [MIT](LICENSE). If you build on the WebDAV implementation, a link back is appreciated.

{zotkit-0.2.0 → zotkit-0.4.0}/README.md

@@ -1,15 +1,21 @@
 # zotkit
 
+[![PyPI](https://img.shields.io/pypi/v/zotkit)](https://pypi.org/project/zotkit/)
+[![Python](https://img.shields.io/pypi/pyversions/zotkit)](https://pypi.org/project/zotkit/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
 **Headless Zotero library management — no desktop app required.**
 
+**English** | [简体中文](README.zh-CN.md)
+
 "Headless" simply means zotkit never needs the Zotero app (or any window) open: it is a
 Python library + CLI that talks straight to the
 [Zotero Web API](https://www.zotero.org/support/dev/web_api/v3/start), so you can
 search, create, tag, and organize items from any terminal — macOS, Windows, or Linux,
-your laptop or a remote server. Uniquely, if your attachments sync to a **personal
-WebDAV server**, zotkit can **upload and download the files themselves** by speaking
-Zotero's WebDAV storage format directly — the one capability the Web API does not
-provide, and (as far as we know) not offered by any other headless tool.
+your laptop or a remote server. If your attachments sync to a **personal WebDAV
+server**, zotkit can **upload and download the files themselves** by speaking
+Zotero's WebDAV storage format directly — a capability the Web API itself does not
+provide. The format is documented in [docs/webdav-format.md](docs/webdav-format.md).
 
 Built for servers, scripts, and **LLM agents**: every write is dry-run by default,
 batched, and version-checked, and you can define a tag taxonomy that is *enforced in
@@ -25,6 +31,18 @@ code* so an agent (or a tired human) can't pollute your library with inconsisten
 | Attachment files on **WebDAV** | ✅ | ⚠️ download at best | ✅ **upload + download** |
 | Tag conventions enforced in code | ❌ | ❌ | ✅ optional `conventions.toml` |
 
+## The zotkit family
+
+Two independent, complementary projects — use either on its own, or both together:
+
+| | What it is |
+|---|---|
+| **zotkit** (this repo) | Headless Python CLI + library. Runs anywhere, needs no Zotero app. |
+| **[zotkit-reader](https://github.com/oldantique/zotkit-reader)** | A Zotero Reader sidebar that embeds a Codex/Claude agent beside the PDF you're reading. Maintained by [@ChanceSiyuan](https://github.com/ChanceSiyuan). |
+
+They share a name and a philosophy, not a codebase: zotkit-reader can call zotkit over MCP
+for library-wide operations, but neither requires the other to be installed.
+
 ## Install
 
 Pure Python (3.11+), no platform-specific bits — the same package works on macOS,
@@ -50,6 +68,9 @@ Copy [`.env.example`](.env.example) to `./.env`, `~/.config/zotkit/env`, or any
   `attach`/`fetch` automatically use Zotero Storage through the Web API's upload/download
   endpoints instead. The storage mode is detected from your `.env`, nothing to configure.
 
+After filling it in, run **`zotkit doctor`** — it validates the config file, API
+access, and attachment storage, and tells you exactly what to fix if anything fails.
+
 Optionally, copy [`conventions.example.toml`](conventions.example.toml) to
 `conventions.toml` next to your `.env` to define a namespaced tag taxonomy
 (`field:physics`, `status:to-read`, …). With it in place, `zotkit create` / `zotkit tag`
@@ -61,7 +82,12 @@ Optionally, copy [`conventions.example.toml`](conventions.example.toml) to
 zotkit find --title "boson sampling"        # search by title/tag/collection
 zotkit find --tag status:to-read
 
-zotkit create --file papers.json            # dry-run: shows what would be created
+zotkit create --arxiv 2401.12345            # fetch arXiv metadata, dry-run preview
+zotkit create --arxiv 2401.12345 --apply --tags field:ai   # create + download & attach the PDF
+zotkit create --doi 10.1038/nature14539     # fetch CrossRef metadata by DOI (no PDF —
+                                            #   usually paywalled; attach manually after)
+
+zotkit create --file papers.json            # batch from JSON: dry-run preview
 zotkit create --file papers.json --apply    # create (dedups by DOI/title)
 zotkit attach --from papers.created.json --all   # upload the PDFs to WebDAV
 
@@ -76,7 +102,14 @@ zotkit backup                               # full JSON snapshot -> backups/
 zotkit lint field:physics topic:new-idea    # offline tag check
 ```
 
-Item JSON for `zotkit create` (a list, one object per reference):
+`--arxiv` takes an id or an abs/pdf URL and maps the full record (all authors,
+abstract, date, DOI, `preprint` item type); `--doi` maps CrossRef records
+(journal articles, conference papers, books, chapters, …) and refuses to guess
+on CrossRef types it doesn't know. Both accept `--collection` and `--tags`, and
+`--no-pdf` skips the arXiv PDF. Fetching metadata from arbitrary web pages is
+out of scope (zotkit stays a daemon-free CLI — no translation-server).
+
+Item JSON for `zotkit create --file` (a list, one object per reference):
 
 ```json
 [{"itemType": "journalArticle", "title": "…",
@@ -140,6 +173,16 @@ Details in [`docs/webdav-format.md`](docs/webdav-format.md).
 The format was determined by interoperability inspection of the author's own library.
 This project is not affiliated with or endorsed by Zotero.
 
+## Limits & roadmap
+
+- `find` currently lists the library client-side — instant for hundreds of items,
+  sluggish for many thousands. Server-side search is planned.
+- Group libraries should work for item operations (untested); WebDAV file sync is
+  personal-libraries-only (a Zotero limitation).
+- `--doi`/`--arxiv` import covers arXiv + CrossRef; DataCite-only DOIs and
+  arbitrary-URL scraping (translation-server territory) are out of scope.
+- Planned: an MCP server wrapper, server-side search.
+
 ## License
 
 [MIT](LICENSE). If you build on the WebDAV implementation, a link back is appreciated.

{zotkit-0.2.0 → zotkit-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "zotkit"
-version = "0.2.0"
+version = "0.4.0"
 description = "Headless Zotero library management: Web API CRUD plus direct WebDAV attachment upload/download — no desktop app required"
 readme = "README.md"
 license = "MIT"

{zotkit-0.2.0 → zotkit-0.4.0}/zotkit/__init__.py

@@ -2,4 +2,4 @@
 from .core import Conventions, TagConventionError, Zot, lint_tags, load_conventions
 
 __all__ = ["Zot", "lint_tags", "load_conventions", "Conventions", "TagConventionError"]
-__version__ = "0.2.0"
+__version__ = "0.4.0"

zotkit-0.4.0/zotkit/cli.py

@@ -0,0 +1,258 @@
+"""zotkit — CLI over zotkit.core.Zot.
+
+Subcommands: doctor, find, create, attach, fetch, tag, status, move, backup, lint.
+Write commands print what they did; `create` is dry-run unless --apply.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from . import __version__
+from .core import Zot, lint_tags, load_conventions, load_env
+
+
+def _print_items(rows):
+    for r in rows:
+        print(f"{r['key']}  [{r['itemType']}]  {r['title'][:75]}")
+        print(f"    collections: {r['collections']}")
+        print(f"    tags: {r['tags']}")
+    print(f"\n{len(rows)} match(es)")
+
+
+def _doctor() -> int:
+    """Validate the whole setup; each failing check explains how to fix it."""
+    import httpx
+    from urllib.parse import urlsplit
+    print(f"zotkit {__version__}")
+    try:
+        env = load_env()
+    except FileNotFoundError as e:
+        print(f"✗ config: {e}")
+        return 1
+    print(f"✓ config: {env['_env_path']}")
+    conv = load_conventions()
+    print(f"✓ conventions: {'enforced' if conv else 'none configured (tags unrestricted)'}")
+    try:
+        z = Zot()
+        access = z.z.key_info().get("access", {}).get("user", {})
+        n = z.z.count_items()
+        print(f"✓ Zotero API: library {env['ZOTERO_LIBRARY_ID']} reachable, {n} items, "
+              f"write access: {bool(access.get('write'))}")
+    except Exception as e:
+        print(f"✗ Zotero API: {e}\n  check ZOTERO_LIBRARY_ID (numeric userID) and "
+              f"ZOTERO_API_KEY at https://www.zotero.org/settings/keys")
+        return 1
+    if "WEBDAV_URL" in env:
+        base, auth = z._webdav()
+        host = urlsplit(base).netloc
+        try:
+            with httpx.Client(auth=auth, timeout=30) as c:
+                r1 = c.put(f"{base}/zotkit-doctor-probe.txt", content=b"zotkit doctor probe")
+                c.delete(f"{base}/zotkit-doctor-probe.txt")
+            if r1.status_code in (200, 201, 204):
+                print(f"✓ WebDAV: {host} writable — attachments use WebDAV")
+            else:
+                print(f"✗ WebDAV: PUT returned {r1.status_code} — check credentials and "
+                      f"that WEBDAV_URL ends with /zotero/")
+                return 1
+        except Exception as e:
+            print(f"✗ WebDAV: {e}")
+            return 1
+    else:
+        print("✓ attachments: Zotero Storage mode (no WEBDAV_* configured)")
+    print("all good ✓")
+    return 0
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="zotkit", description="Headless Zotero library CLI")
+    ap.add_argument("--version", action="version", version=f"zotkit {__version__}")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    sub.add_parser("doctor", help="validate setup: config, API access, attachment storage")
+
+    p = sub.add_parser("find", help="search items by title/tag/collection")
+    p.add_argument("--title"); p.add_argument("--tag"); p.add_argument("--collection")
+
+    p = sub.add_parser("create", help="create items from a JSON file, an arXiv id, "
+                                      "or a DOI (dry unless --apply)")
+    src = p.add_mutually_exclusive_group(required=True)
+    src.add_argument("--file")
+    src.add_argument("--arxiv", metavar="ID_OR_URL",
+                     help="arXiv id (2401.12345) or abs/pdf URL; fetches metadata, "
+                          "and with --apply also downloads + attaches the PDF")
+    src.add_argument("--doi", metavar="DOI",
+                     help="DOI (or doi.org URL); fetches metadata from CrossRef")
+    p.add_argument("--apply", action="store_true")
+    p.add_argument("--no-dedup", action="store_true")
+    p.add_argument("--loose-tags", action="store_true", help="warn instead of error on tag violations")
+    p.add_argument("--no-pdf", action="store_true",
+                   help="with --arxiv --apply: skip downloading/attaching the PDF")
+    p.add_argument("--collection", help="with --arxiv/--doi: file the item here")
+    p.add_argument("--tags", help="with --arxiv/--doi: comma-separated tags, "
+                                  "e.g. field:ml,status:to-read")
+
+    p = sub.add_parser("attach", help="attach a local file to an item (WebDAV or Zotero Storage)")
+    p.add_argument("--key"); p.add_argument("--pdf")
+    p.add_argument("--from", dest="src", help="a .created.json list (batch)")
+    p.add_argument("--all", action="store_true")
+
+    p = sub.add_parser("fetch", help="download attachment files (WebDAV or Zotero Storage)")
+    p.add_argument("--key"); p.add_argument("--title"); p.add_argument("--collection")
+    p.add_argument("--out", default="downloads")
+
+    p = sub.add_parser("tag", help="add/remove tags on an item")
+    p.add_argument("key"); p.add_argument("tags", nargs="+")
+    p.add_argument("--rm", action="store_true", help="remove instead of add")
+
+    p = sub.add_parser("status", help="set reading status (replaces the status: tag)")
+    p.add_argument("key"); p.add_argument("value", help="e.g. to-read / reading / read")
+
+    p = sub.add_parser("move", help="move item to a collection (name or 'Parent :: Child')")
+    p.add_argument("key"); p.add_argument("collection")
+    p.add_argument("--add", action="store_true", help="add as extra home instead of replacing")
+
+    sub.add_parser("backup", help="full JSON backup to backups/ next to your .env")
+
+    p = sub.add_parser("lint", help="check tags against conventions.toml (no API calls)")
+    p.add_argument("tags", nargs="+")
+
+    a = ap.parse_args(argv)
+
+    if a.cmd == "doctor":
+        return _doctor()
+
+    if a.cmd == "lint":
+        conv = load_conventions()
+        if conv is None:
+            print("no conventions configured — add a conventions.toml next to your .env "
+                  "(see README); nothing to check")
+            return 0
+        problems = lint_tags(a.tags, conventions=conv, auto_load=False)
+        print("\n".join(problems) if problems else "OK")
+        return 1 if problems else 0
+
+    zot = Zot()
+
+    if a.cmd == "find":
+        _print_items(zot.find(a.title, a.tag, a.collection))
+
+    elif a.cmd == "create":
+        pdf_url = None
+        if a.file:
+            data = json.load(open(a.file, encoding="utf-8"))
+            items = data["items"] if isinstance(data, dict) else data
+        else:
+            from .metadata import MetadataError, fetch_arxiv, fetch_doi
+            try:
+                if a.arxiv:
+                    item, pdf_url = fetch_arxiv(a.arxiv)
+                else:
+                    item = fetch_doi(a.doi)
+            except MetadataError as e:
+                print(f"error: {e}", file=sys.stderr)
+                return 1
+            if a.collection:
+                item["collection"] = a.collection
+            if a.tags:
+                item["tags"] = [t.strip() for t in a.tags.split(",") if t.strip()]
+            items = [item]
+        if not a.apply:
+            for d in items:
+                problems = lint_tags(d.get("tags", []), conventions=zot.conventions,
+                                     auto_load=False)
+                if not a.file:  # identifier mode: show the full fetched record
+                    print(json.dumps(d, ensure_ascii=False, indent=2))
+                    if problems:
+                        print("  !! " + "; ".join(problems))
+                    continue
+                flag = ("  !! " + "; ".join(problems)) if problems else ""
+                print(f"  [dry] {d.get('collection')} | {d.get('title','')[:60]}{flag}")
+            print(f"{len(items)} item(s). DRY — add --apply to create.")
+            return 0
+        created = zot.create_items(items, dedup=not a.no_dedup, strict_tags=not a.loose_tags)
+        new = [c for c in created if c.get("key")]
+        if a.file:
+            out = Path(a.file).with_suffix(".created.json")
+            json.dump(new, open(out, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
+            print(f"created {len(new)} (skipped {len(created)-len(new)} dup) -> {out}")
+            if any(c.get("file_path") for c in new):
+                print(f"attach PDFs: zotkit attach --from {out} --all")
+            return 0
+        # identifier mode: one item
+        if not new:
+            print("skipped: an item with this DOI/title already exists (use --no-dedup "
+                  "to force)")
+            return 0
+        key = new[0]["key"]
+        print(f"created {key}  {items[0].get('title', '')[:70]}")
+        if pdf_url and not a.no_pdf:
+            from .metadata import download_pdf
+            import tempfile
+            with tempfile.TemporaryDirectory() as td:
+                pdf = Path(td) / (items[0]["archiveID"].replace("arXiv:", "arXiv-")
+                                  .replace("/", "_") + ".pdf")
+                try:
+                    download_pdf(pdf_url, pdf)
+                except MetadataError as e:
+                    print(f"item created, but PDF failed: {e}\n"
+                          f"  attach manually: zotkit attach --key {key} --pdf <file>",
+                          file=sys.stderr)
+                    return 1
+                info = zot.attach(key, pdf)
+            print(f"attached {info['filename']} ({info['storage']})")
+        elif a.doi:
+            print("no PDF downloaded (publisher PDFs are usually paywalled) — "
+                  f"attach one with: zotkit attach --key {key} --pdf <file>")
+
+    elif a.cmd == "attach":
+        if a.key and a.pdf:
+            print(zot.attach(a.key, a.pdf))
+        elif a.src:
+            rows = json.load(open(a.src, encoding="utf-8"))
+            done = 0
+            for m in rows:
+                if not m.get("file_path") or not Path(m["file_path"]).exists():
+                    print(f"  SKIP no file: {str(m.get('title'))[:55]}"); continue
+                if zot.has_attachment(m["key"]):
+                    print(f"  SKIP attached: {str(m.get('title'))[:55]}"); continue
+                if not a.all and done >= 1:
+                    break
+                print(f"- {str(m.get('title'))[:60]}")
+                zot.attach(m["key"], m["file_path"]); done += 1
+            print(f"attached {done}")
+        else:
+            ap.error("attach needs --key + --pdf, or --from <created.json>")
+
+    elif a.cmd == "fetch":
+        keys = [a.key] if a.key else [r["key"] for r in zot.find(a.title, None, a.collection)]
+        n = 0
+        for k in keys:
+            for p_ in zot.fetch(k, a.out):
+                print(f"  saved: {p_}"); n += 1
+        print(f"downloaded {n} file(s) to {a.out}")
+
+    elif a.cmd == "tag":
+        if a.rm:
+            zot.remove_tags(a.key, *a.tags); print(f"removed {a.tags} from {a.key}")
+        else:
+            zot.add_tags(a.key, *a.tags); print(f"added {a.tags} to {a.key}")
+
+    elif a.cmd == "status":
+        zot.set_status(a.key, a.value); print(f"{a.key} -> status:{a.value}")
+
+    elif a.cmd == "move":
+        zot.move(a.key, a.collection, add=a.add)
+        print(f"{a.key} -> '{a.collection}'{' (added)' if a.add else ''}")
+
+    elif a.cmd == "backup":
+        print("backup ->", zot.backup())
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

zotkit-0.4.0/zotkit/metadata.py

@@ -0,0 +1,223 @@
+"""zotkit.metadata — identifier → item-JSON fetchers (arXiv, CrossRef DOI).
+
+Each fetcher returns a plain item dict in the exact shape `zotkit create --file`
+consumes, so the CLI runs every source through the same lint/dry-run/create path.
+Scope is deliberately narrow: stable JSON/Atom APIs only — no translation-server,
+no arbitrary-URL scraping (zotkit stays a daemon-free CLI).
+
+Abstracts are stored verbatim (line breaks, LaTeX, JATS markup untouched).
+"""
+from __future__ import annotations
+
+import re
+import time
+import xml.etree.ElementTree as ET
+from urllib.parse import quote
+
+import httpx
+
+from . import __version__
+
+CONTACT = "mailto:Eric.M.990909@gmail.com"
+USER_AGENT = f"zotkit/{__version__} (https://github.com/oldantique/zotkit; {CONTACT})"
+
+_ATOM = "{http://www.w3.org/2005/Atom}"
+_ARXIV = "{http://arxiv.org/schemas/atom}"
+
+# CrossRef work types with a faithful Zotero equivalent. Anything else errors —
+# guessing an itemType would silently corrupt the record.
+CROSSREF_TYPES = {
+    "journal-article": "journalArticle",
+    "proceedings-article": "conferencePaper",
+    "posted-content": "preprint",
+    "book": "book",
+    "monograph": "book",
+    "edited-book": "book",
+    "reference-book": "book",
+    "book-chapter": "bookSection",
+    "book-section": "bookSection",
+    "book-part": "bookSection",
+    "report": "report",
+    "dissertation": "thesis",
+    "dataset": "dataset",
+    "standard": "standard",
+}
+
+# where CrossRef's container-title belongs, per Zotero itemType
+_CONTAINER_FIELD = {
+    "journalArticle": "publicationTitle",
+    "conferencePaper": "proceedingsTitle",
+    "bookSection": "bookTitle",
+}
+
+
+class MetadataError(RuntimeError):
+    """A metadata lookup failed in a way the user must resolve (bad id,
+    unknown record, unsupported type, network trouble)."""
+
+
+def _get(url: str, **kw) -> httpx.Response:
+    try:
+        r = httpx.get(url, headers={"User-Agent": USER_AGENT}, timeout=30,
+                      follow_redirects=True, **kw)
+        if r.status_code == 429:  # rate-limited: honor Retry-After once, then give up
+            wait = min(int(r.headers.get("retry-after") or 5), 30)
+            time.sleep(wait)
+            r = httpx.get(url, headers={"User-Agent": USER_AGENT}, timeout=30,
+                          follow_redirects=True, **kw)
+            if r.status_code == 429:
+                raise MetadataError(
+                    f"{httpx.URL(url).host} is rate-limiting (HTTP 429) — "
+                    "wait a minute and retry")
+    except httpx.HTTPError as e:
+        raise MetadataError(f"network error talking to {httpx.URL(url).host}: {e}") from e
+    return r
+
+
+def _squash(s: str) -> str:
+    """Collapse whitespace runs — for titles/names, never for abstracts."""
+    return re.sub(r"\s+", " ", s or "").strip()
+
+
+def _person(name: str) -> dict:
+    """Split a display name into first/last on the final space; single-token
+    names go in lastName alone (matches Zotero's single-field mode)."""
+    name = _squash(name)
+    if " " in name:
+        first, last = name.rsplit(" ", 1)
+        return {"creatorType": "author", "firstName": first, "lastName": last}
+    return {"creatorType": "author", "firstName": "", "lastName": name}
+
+
+# ---------- arXiv ----------
+
+_ARXIV_ID = re.compile(r"(?:arxiv\.org/(?:abs|pdf)/|arxiv:)?"
+                       r"((?:\d{4}\.\d{4,5}|[a-z-]+(?:\.[A-Z]{2})?/\d{7})(?:v\d+)?)"
+                       r"(?:\.pdf)?$", re.IGNORECASE)
+
+
+def arxiv_id(id_or_url: str) -> str:
+    """Accept a bare id (2401.12345, 2401.12345v2, math.GT/0309136) or an
+    arxiv.org abs/pdf URL; return the bare id."""
+    s = id_or_url.strip().removeprefix("https://").removeprefix("http://").removeprefix("www.")
+    m = _ARXIV_ID.search(s)
+    if not m:
+        raise MetadataError(f"'{id_or_url}' does not look like an arXiv id or URL")
+    return m.group(1)
+
+
+def fetch_arxiv(id_or_url: str) -> tuple[dict, str]:
+    """arXiv export API → (item dict, pdf url)."""
+    aid = arxiv_id(id_or_url)
+    r = _get("https://export.arxiv.org/api/query", params={"id_list": aid})
+    if r.status_code != 200:
+        raise MetadataError(f"arXiv API returned HTTP {r.status_code} for '{aid}'")
+    try:
+        feed = ET.fromstring(r.text)
+    except ET.ParseError as e:
+        raise MetadataError(f"arXiv API returned unparseable XML: {e}") from e
+    entry = feed.find(f"{_ATOM}entry")
+    title = _squash(entry.findtext(f"{_ATOM}title", "")) if entry is not None else ""
+    if entry is None or not title:
+        raise MetadataError(f"arXiv has no record for '{aid}' — check the id")
+    if title == "Error":
+        raise MetadataError(f"arXiv API error for '{aid}': "
+                            f"{_squash(entry.findtext(f'{_ATOM}summary', ''))}")
+
+    bare = re.sub(r"v\d+$", "", aid)
+    doi = _squash(entry.findtext(f"{_ARXIV}doi", "")) or f"10.48550/arXiv.{bare}"
+    item = {
+        "itemType": "preprint",
+        "title": title,
+        "creators": [_person(n.text or "")
+                     for n in entry.findall(f"{_ATOM}author/{_ATOM}name")],
+        "abstractNote": (entry.findtext(f"{_ATOM}summary", "") or "").strip("\n "),
+        "date": (entry.findtext(f"{_ATOM}published", "") or "")[:10],
+        "url": f"https://arxiv.org/abs/{aid}",
+        "DOI": doi,
+        "repository": "arXiv",
+        "archiveID": f"arXiv:{aid}",
+        "libraryCatalog": "arXiv.org",
+    }
+    return item, f"https://arxiv.org/pdf/{aid}"
+
+
+def download_pdf(url: str, dest) -> None:
+    """Download a PDF to `dest`, refusing anything that isn't actually a PDF."""
+    r = _get(url)
+    if r.status_code != 200:
+        raise MetadataError(f"PDF download failed: HTTP {r.status_code} from {url}")
+    if not r.content.startswith(b"%PDF"):
+        raise MetadataError(f"{url} did not return a PDF "
+                            f"(got {r.headers.get('content-type', 'unknown type')})")
+    with open(dest, "wb") as f:
+        f.write(r.content)
+
+
+# ---------- CrossRef DOI ----------
+
+def _crossref_date(msg: dict) -> str:
+    for k in ("issued", "published-print", "published-online", "created"):
+        parts = (msg.get(k) or {}).get("date-parts", [[]])[0]
+        if parts and parts[0]:
+            return "-".join(f"{p:02d}" if i else str(p) for i, p in enumerate(parts))
+    return ""
+
+
+def fetch_doi(doi: str) -> dict:
+    """CrossRef works API → item dict. Unknown CrossRef types are an error."""
+    doi = re.sub(r"^(https?://(dx\.)?doi\.org/|doi:)", "", doi.strip(), flags=re.IGNORECASE)
+    r = _get(f"https://api.crossref.org/works/{quote(doi, safe='')}")
+    if r.status_code == 404:
+        raise MetadataError(f"CrossRef has no record for DOI '{doi}' — check the DOI "
+                            "(DataCite-only DOIs, e.g. Zenodo/arXiv, are not in CrossRef)")
+    if r.status_code != 200:
+        raise MetadataError(f"CrossRef API returned HTTP {r.status_code} for '{doi}'")
+    msg = r.json()["message"]
+
+    ctype = msg.get("type", "")
+    itype = CROSSREF_TYPES.get(ctype)
+    if not itype:
+        raise MetadataError(
+            f"CrossRef type '{ctype}' has no zotkit mapping — supported: "
+            + ", ".join(sorted(CROSSREF_TYPES)) + ". Create this item via --file instead.")
+
+    creators = []
+    for role, ztype in (("author", "author"), ("editor", "editor")):
+        for a in msg.get(role, []):
+            if a.get("family"):
+                creators.append({"creatorType": ztype,
+                                 "firstName": _squash(a.get("given", "")),
+                                 "lastName": _squash(a["family"])})
+            elif a.get("name"):
+                creators.append({"creatorType": ztype, "name": _squash(a["name"])})
+
+    title = _squash((msg.get("title") or [""])[0])
+    subtitle = _squash((msg.get("subtitle") or [""])[0])
+    if subtitle and subtitle.lower() not in title.lower():
+        title = f"{title}: {subtitle}"
+    item = {
+        "itemType": itype,
+        "title": title,
+        "creators": creators,
+        "abstractNote": msg.get("abstract", ""),
+        "date": _crossref_date(msg),
+        "DOI": doi,
+        "url": msg.get("URL") or f"https://doi.org/{doi}",
+        "volume": msg.get("volume", ""),
+        "issue": msg.get("issue", ""),
+        "pages": msg.get("page", ""),
+        "publisher": _squash(msg.get("publisher", "")),
+        "language": msg.get("language", ""),
+        "ISSN": (msg.get("ISSN") or [""])[0],
+        "ISBN": (msg.get("ISBN") or [""])[0],
+        "journalAbbreviation": _squash((msg.get("short-container-title") or [""])[0]),
+        "libraryCatalog": "CrossRef",
+    }
+    container = _squash((msg.get("container-title") or [""])[0])
+    if container:
+        item[_CONTAINER_FIELD.get(itype, "publicationTitle")] = container
+    if not item["title"]:
+        raise MetadataError(f"CrossRef record for '{doi}' has no title — refusing to "
+                            "create an empty item")
+    return {k: v for k, v in item.items() if v not in ("", [])}

{zotkit-0.2.0 → zotkit-0.4.0/zotkit.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: zotkit
-Version: 0.2.0
+Version: 0.4.0
 Summary: Headless Zotero library management: Web API CRUD plus direct WebDAV attachment upload/download — no desktop app required
 Author: Shawn
 License-Expression: MIT
@@ -20,16 +20,22 @@ Dynamic: license-file
 
 # zotkit
 
+[![PyPI](https://img.shields.io/pypi/v/zotkit)](https://pypi.org/project/zotkit/)
+[![Python](https://img.shields.io/pypi/pyversions/zotkit)](https://pypi.org/project/zotkit/)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
 **Headless Zotero library management — no desktop app required.**
 
+**English** | [简体中文](README.zh-CN.md)
+
 "Headless" simply means zotkit never needs the Zotero app (or any window) open: it is a
 Python library + CLI that talks straight to the
 [Zotero Web API](https://www.zotero.org/support/dev/web_api/v3/start), so you can
 search, create, tag, and organize items from any terminal — macOS, Windows, or Linux,
-your laptop or a remote server. Uniquely, if your attachments sync to a **personal
-WebDAV server**, zotkit can **upload and download the files themselves** by speaking
-Zotero's WebDAV storage format directly — the one capability the Web API does not
-provide, and (as far as we know) not offered by any other headless tool.
+your laptop or a remote server. If your attachments sync to a **personal WebDAV
+server**, zotkit can **upload and download the files themselves** by speaking
+Zotero's WebDAV storage format directly — a capability the Web API itself does not
+provide. The format is documented in [docs/webdav-format.md](docs/webdav-format.md).
 
 Built for servers, scripts, and **LLM agents**: every write is dry-run by default,
 batched, and version-checked, and you can define a tag taxonomy that is *enforced in
@@ -45,6 +51,18 @@ code* so an agent (or a tired human) can't pollute your library with inconsisten
 | Attachment files on **WebDAV** | ✅ | ⚠️ download at best | ✅ **upload + download** |
 | Tag conventions enforced in code | ❌ | ❌ | ✅ optional `conventions.toml` |
 
+## The zotkit family
+
+Two independent, complementary projects — use either on its own, or both together:
+
+| | What it is |
+|---|---|
+| **zotkit** (this repo) | Headless Python CLI + library. Runs anywhere, needs no Zotero app. |
+| **[zotkit-reader](https://github.com/oldantique/zotkit-reader)** | A Zotero Reader sidebar that embeds a Codex/Claude agent beside the PDF you're reading. Maintained by [@ChanceSiyuan](https://github.com/ChanceSiyuan). |
+
+They share a name and a philosophy, not a codebase: zotkit-reader can call zotkit over MCP
+for library-wide operations, but neither requires the other to be installed.
+
 ## Install
 
 Pure Python (3.11+), no platform-specific bits — the same package works on macOS,
@@ -70,6 +88,9 @@ Copy [`.env.example`](.env.example) to `./.env`, `~/.config/zotkit/env`, or any
   `attach`/`fetch` automatically use Zotero Storage through the Web API's upload/download
   endpoints instead. The storage mode is detected from your `.env`, nothing to configure.
 
+After filling it in, run **`zotkit doctor`** — it validates the config file, API
+access, and attachment storage, and tells you exactly what to fix if anything fails.
+
 Optionally, copy [`conventions.example.toml`](conventions.example.toml) to
 `conventions.toml` next to your `.env` to define a namespaced tag taxonomy
 (`field:physics`, `status:to-read`, …). With it in place, `zotkit create` / `zotkit tag`
@@ -81,7 +102,12 @@ Optionally, copy [`conventions.example.toml`](conventions.example.toml) to
 zotkit find --title "boson sampling"        # search by title/tag/collection
 zotkit find --tag status:to-read
 
-zotkit create --file papers.json            # dry-run: shows what would be created
+zotkit create --arxiv 2401.12345            # fetch arXiv metadata, dry-run preview
+zotkit create --arxiv 2401.12345 --apply --tags field:ai   # create + download & attach the PDF
+zotkit create --doi 10.1038/nature14539     # fetch CrossRef metadata by DOI (no PDF —
+                                            #   usually paywalled; attach manually after)
+
+zotkit create --file papers.json            # batch from JSON: dry-run preview
 zotkit create --file papers.json --apply    # create (dedups by DOI/title)
 zotkit attach --from papers.created.json --all   # upload the PDFs to WebDAV
 
@@ -96,7 +122,14 @@ zotkit backup                               # full JSON snapshot -> backups/
 zotkit lint field:physics topic:new-idea    # offline tag check
 ```
 
-Item JSON for `zotkit create` (a list, one object per reference):
+`--arxiv` takes an id or an abs/pdf URL and maps the full record (all authors,
+abstract, date, DOI, `preprint` item type); `--doi` maps CrossRef records
+(journal articles, conference papers, books, chapters, …) and refuses to guess
+on CrossRef types it doesn't know. Both accept `--collection` and `--tags`, and
+`--no-pdf` skips the arXiv PDF. Fetching metadata from arbitrary web pages is
+out of scope (zotkit stays a daemon-free CLI — no translation-server).
+
+Item JSON for `zotkit create --file` (a list, one object per reference):
 
 ```json
 [{"itemType": "journalArticle", "title": "…",
@@ -160,6 +193,16 @@ Details in [`docs/webdav-format.md`](docs/webdav-format.md).
 The format was determined by interoperability inspection of the author's own library.
 This project is not affiliated with or endorsed by Zotero.
 
+## Limits & roadmap
+
+- `find` currently lists the library client-side — instant for hundreds of items,
+  sluggish for many thousands. Server-side search is planned.
+- Group libraries should work for item operations (untested); WebDAV file sync is
+  personal-libraries-only (a Zotero limitation).
+- `--doi`/`--arxiv` import covers arXiv + CrossRef; DataCite-only DOIs and
+  arbitrary-URL scraping (translation-server territory) are out of scope.
+- Planned: an MCP server wrapper, server-side search.
+
 ## License
 
 [MIT](LICENSE). If you build on the WebDAV implementation, a link back is appreciated.

{zotkit-0.2.0 → zotkit-0.4.0}/zotkit.egg-info/SOURCES.txt

@@ -4,6 +4,7 @@ pyproject.toml
 zotkit/__init__.py
 zotkit/cli.py
 zotkit/core.py
+zotkit/metadata.py
 zotkit.egg-info/PKG-INFO
 zotkit.egg-info/SOURCES.txt
 zotkit.egg-info/dependency_links.txt

zotkit-0.2.0/zotkit/cli.py

@@ -1,144 +0,0 @@
-"""zotkit — CLI over zotkit.core.Zot.
-
-Subcommands: find, create, attach, fetch, tag, status, move, backup, lint.
-Write commands print what they did; `create` is dry-run unless --apply.
-"""
-from __future__ import annotations
-
-import argparse
-import json
-import sys
-from pathlib import Path
-
-from .core import Zot, lint_tags, load_conventions
-
-
-def _print_items(rows):
-    for r in rows:
-        print(f"{r['key']}  [{r['itemType']}]  {r['title'][:75]}")
-        print(f"    collections: {r['collections']}")
-        print(f"    tags: {r['tags']}")
-    print(f"\n{len(rows)} match(es)")
-
-
-def main(argv=None):
-    ap = argparse.ArgumentParser(prog="zotkit", description="Headless Zotero library CLI")
-    sub = ap.add_subparsers(dest="cmd", required=True)
-
-    p = sub.add_parser("find", help="search items by title/tag/collection")
-    p.add_argument("--title"); p.add_argument("--tag"); p.add_argument("--collection")
-
-    p = sub.add_parser("create", help="create items from a JSON file (dry unless --apply)")
-    p.add_argument("--file", required=True); p.add_argument("--apply", action="store_true")
-    p.add_argument("--no-dedup", action="store_true")
-    p.add_argument("--loose-tags", action="store_true", help="warn instead of error on tag violations")
-
-    p = sub.add_parser("attach", help="attach a local file (WebDAV) to an item")
-    p.add_argument("--key"); p.add_argument("--pdf")
-    p.add_argument("--from", dest="src", help="a .created.json list (batch)")
-    p.add_argument("--all", action="store_true")
-
-    p = sub.add_parser("fetch", help="download attachment files from WebDAV")
-    p.add_argument("--key"); p.add_argument("--title"); p.add_argument("--collection")
-    p.add_argument("--out", default="downloads")
-
-    p = sub.add_parser("tag", help="add/remove tags on an item")
-    p.add_argument("key"); p.add_argument("tags", nargs="+")
-    p.add_argument("--rm", action="store_true", help="remove instead of add")
-
-    p = sub.add_parser("status", help="set reading status (replaces the status: tag)")
-    p.add_argument("key"); p.add_argument("value", help="e.g. to-read / reading / read")
-
-    p = sub.add_parser("move", help="move item to a collection (name or 'Parent :: Child')")
-    p.add_argument("key"); p.add_argument("collection")
-    p.add_argument("--add", action="store_true", help="add as extra home instead of replacing")
-
-    sub.add_parser("backup", help="full JSON backup to backups/ next to your .env")
-
-    p = sub.add_parser("lint", help="check tags against conventions.toml (no API calls)")
-    p.add_argument("tags", nargs="+")
-
-    a = ap.parse_args(argv)
-
-    if a.cmd == "lint":
-        conv = load_conventions()
-        if conv is None:
-            print("no conventions configured — add a conventions.toml next to your .env "
-                  "(see README); nothing to check")
-            return 0
-        problems = lint_tags(a.tags, conventions=conv, auto_load=False)
-        print("\n".join(problems) if problems else "OK")
-        return 1 if problems else 0
-
-    zot = Zot()
-
-    if a.cmd == "find":
-        _print_items(zot.find(a.title, a.tag, a.collection))
-
-    elif a.cmd == "create":
-        data = json.load(open(a.file, encoding="utf-8"))
-        items = data["items"] if isinstance(data, dict) else data
-        if not a.apply:
-            for d in items:
-                problems = lint_tags(d.get("tags", []), conventions=zot.conventions,
-                                     auto_load=False)
-                flag = ("  !! " + "; ".join(problems)) if problems else ""
-                print(f"  [dry] {d.get('collection')} | {d.get('title','')[:60]}{flag}")
-            print(f"{len(items)} item(s). DRY — add --apply to create.")
-            return 0
-        created = zot.create_items(items, dedup=not a.no_dedup, strict_tags=not a.loose_tags)
-        new = [c for c in created if c.get("key")]
-        out = Path(a.file).with_suffix(".created.json")
-        json.dump(new, open(out, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
-        print(f"created {len(new)} (skipped {len(created)-len(new)} dup) -> {out}")
-        if any(c.get("file_path") for c in new):
-            print(f"attach PDFs: zotkit attach --from {out} --all")
-
-    elif a.cmd == "attach":
-        if a.key and a.pdf:
-            print(zot.attach(a.key, a.pdf))
-        elif a.src:
-            rows = json.load(open(a.src, encoding="utf-8"))
-            done = 0
-            for m in rows:
-                if not m.get("file_path") or not Path(m["file_path"]).exists():
-                    print(f"  SKIP no file: {str(m.get('title'))[:55]}"); continue
-                if zot.has_attachment(m["key"]):
-                    print(f"  SKIP attached: {str(m.get('title'))[:55]}"); continue
-                if not a.all and done >= 1:
-                    break
-                print(f"- {str(m.get('title'))[:60]}")
-                zot.attach(m["key"], m["file_path"]); done += 1
-            print(f"attached {done}")
-        else:
-            ap.error("attach needs --key + --pdf, or --from <created.json>")
-
-    elif a.cmd == "fetch":
-        keys = [a.key] if a.key else [r["key"] for r in zot.find(a.title, None, a.collection)]
-        n = 0
-        for k in keys:
-            for p_ in zot.fetch(k, a.out):
-                print(f"  saved: {p_}"); n += 1
-        print(f"downloaded {n} file(s) to {a.out}")
-
-    elif a.cmd == "tag":
-        if a.rm:
-            zot.remove_tags(a.key, *a.tags); print(f"removed {a.tags} from {a.key}")
-        else:
-            zot.add_tags(a.key, *a.tags); print(f"added {a.tags} to {a.key}")
-
-    elif a.cmd == "status":
-        zot.set_status(a.key, a.value); print(f"{a.key} -> status:{a.value}")
-
-    elif a.cmd == "move":
-        zot.move(a.key, a.collection, add=a.add)
-        print(f"{a.key} -> '{a.collection}'{' (added)' if a.add else ''}")
-
-    elif a.cmd == "backup":
-        print("backup ->", zot.backup())
-
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())