zotkit 0.4.0__tar.gz → 0.4.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {zotkit-0.4.0/zotkit.egg-info → zotkit-0.4.2}/PKG-INFO +24 -7
- {zotkit-0.4.0 → zotkit-0.4.2}/README.md +23 -6
- {zotkit-0.4.0 → zotkit-0.4.2}/pyproject.toml +1 -1
- {zotkit-0.4.0 → zotkit-0.4.2}/zotkit/__init__.py +1 -1
- {zotkit-0.4.0 → zotkit-0.4.2}/zotkit/cli.py +71 -38
- {zotkit-0.4.0 → zotkit-0.4.2}/zotkit/metadata.py +130 -26
- {zotkit-0.4.0 → zotkit-0.4.2/zotkit.egg-info}/PKG-INFO +24 -7
- {zotkit-0.4.0 → zotkit-0.4.2}/LICENSE +0 -0
- {zotkit-0.4.0 → zotkit-0.4.2}/setup.cfg +0 -0
- {zotkit-0.4.0 → zotkit-0.4.2}/zotkit/core.py +0 -0
- {zotkit-0.4.0 → zotkit-0.4.2}/zotkit.egg-info/SOURCES.txt +0 -0
- {zotkit-0.4.0 → zotkit-0.4.2}/zotkit.egg-info/dependency_links.txt +0 -0
- {zotkit-0.4.0 → zotkit-0.4.2}/zotkit.egg-info/entry_points.txt +0 -0
- {zotkit-0.4.0 → zotkit-0.4.2}/zotkit.egg-info/requires.txt +0 -0
- {zotkit-0.4.0 → zotkit-0.4.2}/zotkit.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: zotkit
|
|
3
|
-
Version: 0.4.
|
|
3
|
+
Version: 0.4.2
|
|
4
4
|
Summary: Headless Zotero library management: Web API CRUD plus direct WebDAV attachment upload/download — no desktop app required
|
|
5
5
|
Author: Shawn
|
|
6
6
|
License-Expression: MIT
|
|
@@ -104,6 +104,8 @@ zotkit find --tag status:to-read
|
|
|
104
104
|
|
|
105
105
|
zotkit create --arxiv 2401.12345 # fetch arXiv metadata, dry-run preview
|
|
106
106
|
zotkit create --arxiv 2401.12345 --apply --tags field:ai # create + download & attach the PDF
|
|
107
|
+
zotkit create --arxiv 2401.12345 1706.03762 math/0211159 --apply # batch: one metadata
|
|
108
|
+
# request, PDFs politely spaced ≥3 s apart
|
|
107
109
|
zotkit create --doi 10.1038/nature14539 # fetch CrossRef metadata by DOI (no PDF —
|
|
108
110
|
# usually paywalled; attach manually after)
|
|
109
111
|
|
|
@@ -122,12 +124,27 @@ zotkit backup # full JSON snapshot -> backups/
|
|
|
122
124
|
zotkit lint field:physics topic:new-idea # offline tag check
|
|
123
125
|
```
|
|
124
126
|
|
|
125
|
-
`--arxiv` takes
|
|
126
|
-
abstract, date, DOI
|
|
127
|
-
(journal articles, conference papers, books, chapters, …)
|
|
128
|
-
on CrossRef types it doesn't know.
|
|
129
|
-
|
|
130
|
-
|
|
127
|
+
`--arxiv` takes ids or abs/pdf URLs (several, space- or comma-separated) and maps
|
|
128
|
+
the full record (all authors, abstract, date, DOI); `--doi`
|
|
129
|
+
maps CrossRef records (journal articles, conference papers, books, chapters, …)
|
|
130
|
+
and refuses to guess on CrossRef types it doesn't know.
|
|
131
|
+
|
|
132
|
+
**Version of record**: when arXiv reports a *journal* DOI (the paper was formally
|
|
133
|
+
published — as opposed to arXiv's own `10.48550/*` DataCite DOI), `--arxiv`
|
|
134
|
+
builds the journal record from CrossRef instead of a `preprint`: proper item
|
|
135
|
+
type, venue, volume/pages, formal date. The arXiv identity is kept — `arXiv: <id>`
|
|
136
|
+
goes in Extra, the `url` stays the open-access abs page (the journal link lives in
|
|
137
|
+
the DOI field), the arXiv abstract fills in when CrossRef has none, and the PDF
|
|
138
|
+
still comes from arXiv. If the CrossRef lookup fails, the item falls back to the
|
|
139
|
+
preprint record with a warning rather than failing. Items whose abstract zotkit
|
|
140
|
+
wrote also carry an `abstract-source: arxiv|crossref` line in Extra, naming where
|
|
141
|
+
it actually came from. Both accept `--collection`
|
|
142
|
+
and `--tags`, and `--no-pdf` skips the arXiv PDFs. Rate limiting is built into the
|
|
143
|
+
request layer — batches use one arXiv metadata request and space PDF downloads
|
|
144
|
+
per arXiv's terms of use, so callers (humans or agents) never pace themselves. A
|
|
145
|
+
bad id fails alone, not the batch; the exit code is non-zero only if something
|
|
146
|
+
failed. Fetching metadata from arbitrary web pages is out of scope (zotkit stays
|
|
147
|
+
a daemon-free CLI — no translation-server).
|
|
131
148
|
|
|
132
149
|
Item JSON for `zotkit create --file` (a list, one object per reference):
|
|
133
150
|
|
|
@@ -84,6 +84,8 @@ zotkit find --tag status:to-read
|
|
|
84
84
|
|
|
85
85
|
zotkit create --arxiv 2401.12345 # fetch arXiv metadata, dry-run preview
|
|
86
86
|
zotkit create --arxiv 2401.12345 --apply --tags field:ai # create + download & attach the PDF
|
|
87
|
+
zotkit create --arxiv 2401.12345 1706.03762 math/0211159 --apply # batch: one metadata
|
|
88
|
+
# request, PDFs politely spaced ≥3 s apart
|
|
87
89
|
zotkit create --doi 10.1038/nature14539 # fetch CrossRef metadata by DOI (no PDF —
|
|
88
90
|
# usually paywalled; attach manually after)
|
|
89
91
|
|
|
@@ -102,12 +104,27 @@ zotkit backup # full JSON snapshot -> backups/
|
|
|
102
104
|
zotkit lint field:physics topic:new-idea # offline tag check
|
|
103
105
|
```
|
|
104
106
|
|
|
105
|
-
`--arxiv` takes
|
|
106
|
-
abstract, date, DOI
|
|
107
|
-
(journal articles, conference papers, books, chapters, …)
|
|
108
|
-
on CrossRef types it doesn't know.
|
|
109
|
-
|
|
110
|
-
|
|
107
|
+
`--arxiv` takes ids or abs/pdf URLs (several, space- or comma-separated) and maps
|
|
108
|
+
the full record (all authors, abstract, date, DOI); `--doi`
|
|
109
|
+
maps CrossRef records (journal articles, conference papers, books, chapters, …)
|
|
110
|
+
and refuses to guess on CrossRef types it doesn't know.
|
|
111
|
+
|
|
112
|
+
**Version of record**: when arXiv reports a *journal* DOI (the paper was formally
|
|
113
|
+
published — as opposed to arXiv's own `10.48550/*` DataCite DOI), `--arxiv`
|
|
114
|
+
builds the journal record from CrossRef instead of a `preprint`: proper item
|
|
115
|
+
type, venue, volume/pages, formal date. The arXiv identity is kept — `arXiv: <id>`
|
|
116
|
+
goes in Extra, the `url` stays the open-access abs page (the journal link lives in
|
|
117
|
+
the DOI field), the arXiv abstract fills in when CrossRef has none, and the PDF
|
|
118
|
+
still comes from arXiv. If the CrossRef lookup fails, the item falls back to the
|
|
119
|
+
preprint record with a warning rather than failing. Items whose abstract zotkit
|
|
120
|
+
wrote also carry an `abstract-source: arxiv|crossref` line in Extra, naming where
|
|
121
|
+
it actually came from. Both accept `--collection`
|
|
122
|
+
and `--tags`, and `--no-pdf` skips the arXiv PDFs. Rate limiting is built into the
|
|
123
|
+
request layer — batches use one arXiv metadata request and space PDF downloads
|
|
124
|
+
per arXiv's terms of use, so callers (humans or agents) never pace themselves. A
|
|
125
|
+
bad id fails alone, not the batch; the exit code is non-zero only if something
|
|
126
|
+
failed. Fetching metadata from arbitrary web pages is out of scope (zotkit stays
|
|
127
|
+
a daemon-free CLI — no translation-server).
|
|
111
128
|
|
|
112
129
|
Item JSON for `zotkit create --file` (a list, one object per reference):
|
|
113
130
|
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "zotkit"
|
|
7
|
-
version = "0.4.
|
|
7
|
+
version = "0.4.2"
|
|
8
8
|
description = "Headless Zotero library management: Web API CRUD plus direct WebDAV attachment upload/download — no desktop app required"
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
license = "MIT"
|
|
@@ -81,11 +81,13 @@ def main(argv=None):
|
|
|
81
81
|
"or a DOI (dry unless --apply)")
|
|
82
82
|
src = p.add_mutually_exclusive_group(required=True)
|
|
83
83
|
src.add_argument("--file")
|
|
84
|
-
src.add_argument("--arxiv", metavar="ID_OR_URL",
|
|
85
|
-
help="arXiv
|
|
86
|
-
"
|
|
87
|
-
|
|
88
|
-
|
|
84
|
+
src.add_argument("--arxiv", metavar="ID_OR_URL", nargs="+",
|
|
85
|
+
help="arXiv ids (2401.12345) or abs/pdf URLs — several allowed, "
|
|
86
|
+
"space- or comma-separated; fetches metadata in one batched "
|
|
87
|
+
"request, and with --apply also downloads + attaches the PDFs")
|
|
88
|
+
src.add_argument("--doi", metavar="DOI", nargs="+",
|
|
89
|
+
help="DOIs (or doi.org URLs) — several allowed, space- or "
|
|
90
|
+
"comma-separated; fetches metadata from CrossRef")
|
|
89
91
|
p.add_argument("--apply", action="store_true")
|
|
90
92
|
p.add_argument("--no-dedup", action="store_true")
|
|
91
93
|
p.add_argument("--loose-tags", action="store_true", help="warn instead of error on tag violations")
|
|
@@ -141,25 +143,47 @@ def main(argv=None):
|
|
|
141
143
|
_print_items(zot.find(a.title, a.tag, a.collection))
|
|
142
144
|
|
|
143
145
|
elif a.cmd == "create":
|
|
144
|
-
|
|
146
|
+
pdfs: dict[str, tuple[str, str]] = {} # title -> (arXiv id, pdf url)
|
|
147
|
+
failures: list[tuple[str, str]] = [] # (identifier, why)
|
|
145
148
|
if a.file:
|
|
146
149
|
data = json.load(open(a.file, encoding="utf-8"))
|
|
147
150
|
items = data["items"] if isinstance(data, dict) else data
|
|
148
151
|
else:
|
|
149
|
-
from .metadata import MetadataError,
|
|
152
|
+
from .metadata import MetadataError, fetch_arxiv_batch, fetch_doi
|
|
153
|
+
wanted = [w.strip() for arg in (a.arxiv or a.doi)
|
|
154
|
+
for w in arg.split(",") if w.strip()]
|
|
155
|
+
items = []
|
|
150
156
|
try:
|
|
151
157
|
if a.arxiv:
|
|
152
|
-
|
|
158
|
+
for r in fetch_arxiv_batch(wanted):
|
|
159
|
+
if r.get("error"):
|
|
160
|
+
failures.append((r["id"], r["error"]))
|
|
161
|
+
continue
|
|
162
|
+
if r.get("note"): # version-of-record upgrade happened
|
|
163
|
+
print(r["note"])
|
|
164
|
+
if r.get("warning"):
|
|
165
|
+
print(f"warning: {r['warning']}", file=sys.stderr)
|
|
166
|
+
items.append(r["item"])
|
|
167
|
+
pdfs[r["item"]["title"]] = (r["id"], r["pdf_url"])
|
|
153
168
|
else:
|
|
154
|
-
|
|
169
|
+
for d in wanted:
|
|
170
|
+
try:
|
|
171
|
+
items.append(fetch_doi(d))
|
|
172
|
+
except MetadataError as e:
|
|
173
|
+
failures.append((d, str(e)))
|
|
155
174
|
except MetadataError as e:
|
|
156
175
|
print(f"error: {e}", file=sys.stderr)
|
|
157
176
|
return 1
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
177
|
+
for ident, why in failures:
|
|
178
|
+
prefix = f"{ident}: " if len(wanted) > 1 else ""
|
|
179
|
+
print(f"error: {prefix}{why}", file=sys.stderr)
|
|
180
|
+
for item in items:
|
|
181
|
+
if a.collection:
|
|
182
|
+
item["collection"] = a.collection
|
|
183
|
+
if a.tags:
|
|
184
|
+
item["tags"] = [t.strip() for t in a.tags.split(",") if t.strip()]
|
|
185
|
+
if not items:
|
|
186
|
+
return 1
|
|
163
187
|
if not a.apply:
|
|
164
188
|
for d in items:
|
|
165
189
|
problems = lint_tags(d.get("tags", []), conventions=zot.conventions,
|
|
@@ -171,42 +195,51 @@ def main(argv=None):
|
|
|
171
195
|
continue
|
|
172
196
|
flag = (" !! " + "; ".join(problems)) if problems else ""
|
|
173
197
|
print(f" [dry] {d.get('collection')} | {d.get('title','')[:60]}{flag}")
|
|
174
|
-
|
|
175
|
-
|
|
198
|
+
tail = f", {len(failures)} failed to fetch" if failures else ""
|
|
199
|
+
print(f"{len(items)} item(s){tail}. DRY — add --apply to create.")
|
|
200
|
+
return 1 if failures else 0
|
|
176
201
|
created = zot.create_items(items, dedup=not a.no_dedup, strict_tags=not a.loose_tags)
|
|
177
202
|
new = [c for c in created if c.get("key")]
|
|
203
|
+
dup = len(created) - len(new)
|
|
178
204
|
if a.file:
|
|
179
205
|
out = Path(a.file).with_suffix(".created.json")
|
|
180
206
|
json.dump(new, open(out, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
|
|
181
|
-
print(f"created {len(new)} (skipped {
|
|
207
|
+
print(f"created {len(new)} (skipped {dup} dup) -> {out}")
|
|
182
208
|
if any(c.get("file_path") for c in new):
|
|
183
209
|
print(f"attach PDFs: zotkit attach --from {out} --all")
|
|
184
210
|
return 0
|
|
185
|
-
# identifier mode
|
|
186
|
-
if not new:
|
|
211
|
+
# identifier mode
|
|
212
|
+
if not new and not failures:
|
|
187
213
|
print("skipped: an item with this DOI/title already exists (use --no-dedup "
|
|
188
214
|
"to force)")
|
|
189
215
|
return 0
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
216
|
+
for c in new:
|
|
217
|
+
key = c["key"]
|
|
218
|
+
print(f"created {key} {str(c.get('title', ''))[:70]}")
|
|
219
|
+
aid_pdf = pdfs.get(c.get("title"))
|
|
220
|
+
if aid_pdf and not a.no_pdf:
|
|
221
|
+
from .metadata import download_pdf
|
|
222
|
+
import tempfile
|
|
223
|
+
aid, pdf_url = aid_pdf
|
|
224
|
+
with tempfile.TemporaryDirectory() as td:
|
|
225
|
+
pdf = Path(td) / f"arXiv-{aid.replace('/', '_')}.pdf"
|
|
226
|
+
try:
|
|
227
|
+
download_pdf(pdf_url, pdf) # spaced ≥3 s apart by metadata._get
|
|
228
|
+
except MetadataError as e:
|
|
229
|
+
failures.append((aid, f"item {key} created, but PDF failed: {e}"))
|
|
230
|
+
print(f"error: item {key} created, but PDF failed: {e}\n"
|
|
231
|
+
f" attach manually: zotkit attach --key {key} --pdf <file>",
|
|
232
|
+
file=sys.stderr)
|
|
233
|
+
continue
|
|
234
|
+
info = zot.attach(key, pdf)
|
|
235
|
+
print(f"attached {info['filename']} ({info['storage']})")
|
|
236
|
+
if a.doi and new:
|
|
237
|
+
hint = new[0]["key"] if len(new) == 1 else "<key>"
|
|
208
238
|
print("no PDF downloaded (publisher PDFs are usually paywalled) — "
|
|
209
|
-
f"attach one with: zotkit attach --key {
|
|
239
|
+
f"attach one with: zotkit attach --key {hint} --pdf <file>")
|
|
240
|
+
if len(wanted) > 1:
|
|
241
|
+
print(f"batch: created {len(new)}, skipped {dup} dup, failed {len(failures)}")
|
|
242
|
+
return 1 if failures else 0
|
|
210
243
|
|
|
211
244
|
elif a.cmd == "attach":
|
|
212
245
|
if a.key and a.pdf:
|
|
@@ -6,6 +6,11 @@ Scope is deliberately narrow: stable JSON/Atom APIs only — no translation-serv
|
|
|
6
6
|
no arbitrary-URL scraping (zotkit stays a daemon-free CLI).
|
|
7
7
|
|
|
8
8
|
Abstracts are stored verbatim (line breaks, LaTeX, JATS markup untouched).
|
|
9
|
+
|
|
10
|
+
Rate limiting lives here, in the request layer: every request through `_get`
|
|
11
|
+
waits out the per-host minimum interval (arXiv's terms: 1 request / 3 s on a
|
|
12
|
+
single connection; CrossRef polite pool: 1 s). Callers — CLI users, agents,
|
|
13
|
+
batch loops — never need to pace themselves.
|
|
9
14
|
"""
|
|
10
15
|
from __future__ import annotations
|
|
11
16
|
|
|
@@ -56,7 +61,17 @@ class MetadataError(RuntimeError):
|
|
|
56
61
|
unknown record, unsupported type, network trouble)."""
|
|
57
62
|
|
|
58
63
|
|
|
64
|
+
# minimum seconds between requests to the same host (arXiv terms of use: 3 s,
|
|
65
|
+
# covering the export API and PDF downloads; CrossRef polite pool: 1 s)
|
|
66
|
+
_MIN_INTERVAL = {"export.arxiv.org": 3.0, "arxiv.org": 3.0, "api.crossref.org": 1.0}
|
|
67
|
+
_next_ok: dict[str, float] = {}
|
|
68
|
+
|
|
69
|
+
|
|
59
70
|
def _get(url: str, **kw) -> httpx.Response:
|
|
71
|
+
host = httpx.URL(url).host
|
|
72
|
+
wait = _next_ok.get(host, 0.0) - time.monotonic()
|
|
73
|
+
if wait > 0:
|
|
74
|
+
time.sleep(wait)
|
|
60
75
|
try:
|
|
61
76
|
r = httpx.get(url, headers={"User-Agent": USER_AGENT}, timeout=30,
|
|
62
77
|
follow_redirects=True, **kw)
|
|
@@ -67,10 +82,11 @@ def _get(url: str, **kw) -> httpx.Response:
|
|
|
67
82
|
follow_redirects=True, **kw)
|
|
68
83
|
if r.status_code == 429:
|
|
69
84
|
raise MetadataError(
|
|
70
|
-
f"{
|
|
71
|
-
"wait a minute and retry")
|
|
85
|
+
f"{host} is rate-limiting (HTTP 429) — wait a minute and retry")
|
|
72
86
|
except httpx.HTTPError as e:
|
|
73
|
-
raise MetadataError(f"network error talking to {
|
|
87
|
+
raise MetadataError(f"network error talking to {host}: {e}") from e
|
|
88
|
+
finally:
|
|
89
|
+
_next_ok[host] = time.monotonic() + _MIN_INTERVAL.get(host, 0.0)
|
|
74
90
|
return r
|
|
75
91
|
|
|
76
92
|
|
|
@@ -79,6 +95,11 @@ def _squash(s: str) -> str:
|
|
|
79
95
|
return re.sub(r"\s+", " ", s or "").strip()
|
|
80
96
|
|
|
81
97
|
|
|
98
|
+
def _add_extra(item: dict, line: str) -> None:
|
|
99
|
+
"""Append a line to the item's Extra field, never overwriting what's there."""
|
|
100
|
+
item["extra"] = f"{item['extra']}\n{line}" if item.get("extra") else line
|
|
101
|
+
|
|
102
|
+
|
|
82
103
|
def _person(name: str) -> dict:
|
|
83
104
|
"""Split a display name into first/last on the final space; single-token
|
|
84
105
|
names go in lastName alone (matches Zotero's single-field mode)."""
|
|
@@ -106,29 +127,19 @@ def arxiv_id(id_or_url: str) -> str:
|
|
|
106
127
|
return m.group(1)
|
|
107
128
|
|
|
108
129
|
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
entry = feed.find(f"{_ATOM}entry")
|
|
120
|
-
title = _squash(entry.findtext(f"{_ATOM}title", "")) if entry is not None else ""
|
|
121
|
-
if entry is None or not title:
|
|
122
|
-
raise MetadataError(f"arXiv has no record for '{aid}' — check the id")
|
|
123
|
-
if title == "Error":
|
|
124
|
-
raise MetadataError(f"arXiv API error for '{aid}': "
|
|
125
|
-
f"{_squash(entry.findtext(f'{_ATOM}summary', ''))}")
|
|
126
|
-
|
|
127
|
-
bare = re.sub(r"v\d+$", "", aid)
|
|
128
|
-
doi = _squash(entry.findtext(f"{_ARXIV}doi", "")) or f"10.48550/arXiv.{bare}"
|
|
130
|
+
_ARXIV_CHUNK = 50 # ids per id_list request; chunks are spaced out by _get
|
|
131
|
+
|
|
132
|
+
|
|
133
|
+
def _bare(aid: str) -> str:
|
|
134
|
+
return re.sub(r"v\d+$", "", aid, flags=re.IGNORECASE)
|
|
135
|
+
|
|
136
|
+
|
|
137
|
+
def _arxiv_item(entry, aid: str) -> tuple[dict, str]:
|
|
138
|
+
doi = (_squash(entry.findtext(f"{_ARXIV}doi", ""))
|
|
139
|
+
or f"10.48550/arXiv.{_bare(aid)}")
|
|
129
140
|
item = {
|
|
130
141
|
"itemType": "preprint",
|
|
131
|
-
"title": title,
|
|
142
|
+
"title": _squash(entry.findtext(f"{_ATOM}title", "")),
|
|
132
143
|
"creators": [_person(n.text or "")
|
|
133
144
|
for n in entry.findall(f"{_ATOM}author/{_ATOM}name")],
|
|
134
145
|
"abstractNote": (entry.findtext(f"{_ATOM}summary", "") or "").strip("\n "),
|
|
@@ -139,9 +150,99 @@ def fetch_arxiv(id_or_url: str) -> tuple[dict, str]:
|
|
|
139
150
|
"archiveID": f"arXiv:{aid}",
|
|
140
151
|
"libraryCatalog": "arXiv.org",
|
|
141
152
|
}
|
|
153
|
+
if item["abstractNote"]:
|
|
154
|
+
_add_extra(item, "abstract-source: arxiv")
|
|
142
155
|
return item, f"https://arxiv.org/pdf/{aid}"
|
|
143
156
|
|
|
144
157
|
|
|
158
|
+
_DATACITE_PREFIX = "10.48550/" # arXiv's own DOIs; anything else is a journal DOI
|
|
159
|
+
|
|
160
|
+
|
|
161
|
+
def _journal_record(pre: dict, aid: str) -> dict:
|
|
162
|
+
"""Version-of-record upgrade: the arXiv record carries a journal DOI, so
|
|
163
|
+
build the item from CrossRef instead (venue/volume/pages/formal date),
|
|
164
|
+
keeping the arXiv identity: abstract fallback when CrossRef has none,
|
|
165
|
+
`arXiv: <id>` in Extra, and the open-access abs page as url (the journal
|
|
166
|
+
link is carried by the DOI field). Raises MetadataError on lookup failure —
|
|
167
|
+
the caller falls back to the preprint record."""
|
|
168
|
+
j = fetch_doi(pre["DOI"])
|
|
169
|
+
if not j.get("abstractNote") and pre.get("abstractNote"):
|
|
170
|
+
j["abstractNote"] = pre["abstractNote"]
|
|
171
|
+
_add_extra(j, "abstract-source: arxiv")
|
|
172
|
+
j["url"] = pre["url"]
|
|
173
|
+
_add_extra(j, f"arXiv: {aid}")
|
|
174
|
+
return j
|
|
175
|
+
|
|
176
|
+
|
|
177
|
+
def fetch_arxiv_batch(ids_or_urls: list[str]) -> list[dict]:
|
|
178
|
+
"""arXiv export API, one id_list request per ≤50 ids → results in input
|
|
179
|
+
order, each {"id", "item", "pdf_url"} or {"id", "error"}. A bad id never
|
|
180
|
+
fails the batch; only transport-level trouble raises MetadataError.
|
|
181
|
+
|
|
182
|
+
Version of record: when arXiv reports a journal DOI (not its own
|
|
183
|
+
10.48550/* DataCite DOI) the item is rebuilt from CrossRef ("note" key
|
|
184
|
+
says so); if that lookup fails the preprint record stands ("warning" key).
|
|
185
|
+
|
|
186
|
+
The response can't be trusted positionally: entries come back in arbitrary
|
|
187
|
+
order (and with resolved version numbers), so they are mapped back to the
|
|
188
|
+
requested ids by version-stripped id.
|
|
189
|
+
"""
|
|
190
|
+
results: list[dict] = []
|
|
191
|
+
valid: list[dict] = [] # results entries still awaiting an API answer
|
|
192
|
+
for s in ids_or_urls:
|
|
193
|
+
try:
|
|
194
|
+
results.append({"id": arxiv_id(s)})
|
|
195
|
+
valid.append(results[-1])
|
|
196
|
+
except MetadataError as e:
|
|
197
|
+
results.append({"id": s, "error": str(e)})
|
|
198
|
+
|
|
199
|
+
for i in range(0, len(valid), _ARXIV_CHUNK):
|
|
200
|
+
chunk = valid[i:i + _ARXIV_CHUNK]
|
|
201
|
+
# max_results defaults to 10 — without it, batches >10 silently truncate
|
|
202
|
+
r = _get("https://export.arxiv.org/api/query",
|
|
203
|
+
params={"id_list": ",".join(c["id"] for c in chunk),
|
|
204
|
+
"max_results": len(chunk)})
|
|
205
|
+
if r.status_code != 200:
|
|
206
|
+
raise MetadataError(f"arXiv API returned HTTP {r.status_code}")
|
|
207
|
+
try:
|
|
208
|
+
feed = ET.fromstring(r.text)
|
|
209
|
+
except ET.ParseError as e:
|
|
210
|
+
raise MetadataError(f"arXiv API returned unparseable XML: {e}") from e
|
|
211
|
+
emap = {}
|
|
212
|
+
for entry in feed.findall(f"{_ATOM}entry"):
|
|
213
|
+
title = _squash(entry.findtext(f"{_ATOM}title", ""))
|
|
214
|
+
if not title or title == "Error": # error/placeholder entries: unmappable
|
|
215
|
+
continue
|
|
216
|
+
try:
|
|
217
|
+
emap[_bare(arxiv_id(entry.findtext(f"{_ATOM}id", "")))] = entry
|
|
218
|
+
except MetadataError:
|
|
219
|
+
continue
|
|
220
|
+
for c in chunk:
|
|
221
|
+
entry = emap.get(_bare(c["id"]))
|
|
222
|
+
if entry is None:
|
|
223
|
+
c["error"] = f"arXiv has no record for '{c['id']}' — check the id"
|
|
224
|
+
continue
|
|
225
|
+
c["item"], c["pdf_url"] = _arxiv_item(entry, c["id"])
|
|
226
|
+
doi = c["item"]["DOI"]
|
|
227
|
+
if not doi.startswith(_DATACITE_PREFIX):
|
|
228
|
+
try:
|
|
229
|
+
c["item"] = _journal_record(c["item"], c["id"])
|
|
230
|
+
c["note"] = (f"{c['id']} → journal DOI {doi}, "
|
|
231
|
+
"building journal record")
|
|
232
|
+
except MetadataError as e:
|
|
233
|
+
c["warning"] = (f"{c['id']}: journal DOI {doi} lookup failed "
|
|
234
|
+
f"({e}) — keeping preprint record")
|
|
235
|
+
return results
|
|
236
|
+
|
|
237
|
+
|
|
238
|
+
def fetch_arxiv(id_or_url: str) -> tuple[dict, str]:
|
|
239
|
+
"""arXiv export API → (item dict, pdf url); raises on any failure."""
|
|
240
|
+
res = fetch_arxiv_batch([id_or_url])[0]
|
|
241
|
+
if res.get("error"):
|
|
242
|
+
raise MetadataError(res["error"])
|
|
243
|
+
return res["item"], res["pdf_url"]
|
|
244
|
+
|
|
245
|
+
|
|
145
246
|
def download_pdf(url: str, dest) -> None:
|
|
146
247
|
"""Download a PDF to `dest`, refusing anything that isn't actually a PDF."""
|
|
147
248
|
r = _get(url)
|
|
@@ -206,7 +307,7 @@ def fetch_doi(doi: str) -> dict:
|
|
|
206
307
|
"url": msg.get("URL") or f"https://doi.org/{doi}",
|
|
207
308
|
"volume": msg.get("volume", ""),
|
|
208
309
|
"issue": msg.get("issue", ""),
|
|
209
|
-
"pages": msg.get("page", ""),
|
|
310
|
+
"pages": msg.get("page") or msg.get("article-number", ""),
|
|
210
311
|
"publisher": _squash(msg.get("publisher", "")),
|
|
211
312
|
"language": msg.get("language", ""),
|
|
212
313
|
"ISSN": (msg.get("ISSN") or [""])[0],
|
|
@@ -220,4 +321,7 @@ def fetch_doi(doi: str) -> dict:
|
|
|
220
321
|
if not item["title"]:
|
|
221
322
|
raise MetadataError(f"CrossRef record for '{doi}' has no title — refusing to "
|
|
222
323
|
"create an empty item")
|
|
223
|
-
|
|
324
|
+
item = {k: v for k, v in item.items() if v not in ("", [])}
|
|
325
|
+
if item.get("abstractNote"):
|
|
326
|
+
_add_extra(item, "abstract-source: crossref")
|
|
327
|
+
return item
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: zotkit
|
|
3
|
-
Version: 0.4.
|
|
3
|
+
Version: 0.4.2
|
|
4
4
|
Summary: Headless Zotero library management: Web API CRUD plus direct WebDAV attachment upload/download — no desktop app required
|
|
5
5
|
Author: Shawn
|
|
6
6
|
License-Expression: MIT
|
|
@@ -104,6 +104,8 @@ zotkit find --tag status:to-read
|
|
|
104
104
|
|
|
105
105
|
zotkit create --arxiv 2401.12345 # fetch arXiv metadata, dry-run preview
|
|
106
106
|
zotkit create --arxiv 2401.12345 --apply --tags field:ai # create + download & attach the PDF
|
|
107
|
+
zotkit create --arxiv 2401.12345 1706.03762 math/0211159 --apply # batch: one metadata
|
|
108
|
+
# request, PDFs politely spaced ≥3 s apart
|
|
107
109
|
zotkit create --doi 10.1038/nature14539 # fetch CrossRef metadata by DOI (no PDF —
|
|
108
110
|
# usually paywalled; attach manually after)
|
|
109
111
|
|
|
@@ -122,12 +124,27 @@ zotkit backup # full JSON snapshot -> backups/
|
|
|
122
124
|
zotkit lint field:physics topic:new-idea # offline tag check
|
|
123
125
|
```
|
|
124
126
|
|
|
125
|
-
`--arxiv` takes
|
|
126
|
-
abstract, date, DOI
|
|
127
|
-
(journal articles, conference papers, books, chapters, …)
|
|
128
|
-
on CrossRef types it doesn't know.
|
|
129
|
-
|
|
130
|
-
|
|
127
|
+
`--arxiv` takes ids or abs/pdf URLs (several, space- or comma-separated) and maps
|
|
128
|
+
the full record (all authors, abstract, date, DOI); `--doi`
|
|
129
|
+
maps CrossRef records (journal articles, conference papers, books, chapters, …)
|
|
130
|
+
and refuses to guess on CrossRef types it doesn't know.
|
|
131
|
+
|
|
132
|
+
**Version of record**: when arXiv reports a *journal* DOI (the paper was formally
|
|
133
|
+
published — as opposed to arXiv's own `10.48550/*` DataCite DOI), `--arxiv`
|
|
134
|
+
builds the journal record from CrossRef instead of a `preprint`: proper item
|
|
135
|
+
type, venue, volume/pages, formal date. The arXiv identity is kept — `arXiv: <id>`
|
|
136
|
+
goes in Extra, the `url` stays the open-access abs page (the journal link lives in
|
|
137
|
+
the DOI field), the arXiv abstract fills in when CrossRef has none, and the PDF
|
|
138
|
+
still comes from arXiv. If the CrossRef lookup fails, the item falls back to the
|
|
139
|
+
preprint record with a warning rather than failing. Items whose abstract zotkit
|
|
140
|
+
wrote also carry an `abstract-source: arxiv|crossref` line in Extra, naming where
|
|
141
|
+
it actually came from. Both accept `--collection`
|
|
142
|
+
and `--tags`, and `--no-pdf` skips the arXiv PDFs. Rate limiting is built into the
|
|
143
|
+
request layer — batches use one arXiv metadata request and space PDF downloads
|
|
144
|
+
per arXiv's terms of use, so callers (humans or agents) never pace themselves. A
|
|
145
|
+
bad id fails alone, not the batch; the exit code is non-zero only if something
|
|
146
|
+
failed. Fetching metadata from arbitrary web pages is out of scope (zotkit stays
|
|
147
|
+
a daemon-free CLI — no translation-server).
|
|
131
148
|
|
|
132
149
|
Item JSON for `zotkit create --file` (a list, one object per reference):
|
|
133
150
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|