psyclaw 0.27.22 → 0.28.0

package/vendor/ars/scripts/adapters/README.md

@@ -0,0 +1,89 @@
+# ARS Literature Corpus Adapters
+
+Reference adapters that produce `literature_corpus[]` entries for the ARS Material Passport. Defined by the adapter contract at [`academic-pipeline/references/adapters/overview.md`](../../academic-pipeline/references/adapters/overview.md).
+
+## What these are
+
+Three starting-point adapters:
+
+- **`folder_scan.py`** — scans a directory of files, parses citation metadata from filenames.
+- **`zotero.py`** — reads a Better BibTeX JSON export.
+- **`obsidian.py`** — reads an Obsidian vault (frontmatter Convention A or Karpathy-style Convention B).
+
+Each writes two files: a `passport.yaml` with `literature_corpus[]` entries, and a `rejection_log.yaml` listing whatever the adapter could not map onto a valid entry.
+
+**These are references, not products.** If your corpus is in Zotero, Obsidian, or a plain folder, you can run these directly; if it is in Notion, Readwise, Airtable, a custom SQLite, etc., copy a reference adapter and adapt it to your source.
+
+## Running a reference adapter
+
+### folder_scan
+
+```bash
+python scripts/adapters/folder_scan.py \
+    --input /path/to/your/pdf/folder \
+    --passport /tmp/passport.yaml \
+    --rejection-log /tmp/rejection_log.yaml
+```
+
+Filenames are parsed with two conventions plus a fallback: `{Family}_{Year}_{title}.{ext}`, `{Family}{Year}{title}.{ext}`, or `... {Year} ... {Family} ...`. Files whose filename cannot yield both a family and a year are rejected. Non-ASCII filenames are rejected (Unicode support is a deferred extension). Symlinks are followed but the rejection log records the un-resolved path so collisions across subdirectories remain distinguishable.
+
+### zotero
+
+Export your Zotero library as **Better BibTeX JSON** (Zotero → File → Export Library → Format: Better CSL JSON, with the Better BibTeX extension installed). Then:
+
+```bash
+python scripts/adapters/zotero.py \
+    --input ~/zotero_export.json \
+    --passport /tmp/passport.yaml \
+    --rejection-log /tmp/rejection_log.yaml
+```
+
+This adapter does NOT call the Zotero Web API. If you want live sync, write your own adapter using this file as a starting point. Items missing required fields (`title`, `authors`, parseable `year`, `source_pointer`) are rejected with a categorical reason; seasonal dates like `Spring 2024` are rejected as `year_unparseable`.
+
+### obsidian
+
+```bash
+python scripts/adapters/obsidian.py \
+    --input ~/ObsidianVault \
+    --passport /tmp/passport.yaml \
+    --rejection-log /tmp/rejection_log.yaml
+```
+
+Files under `_templates/` and `.obsidian/` are skipped. Wikilinks (`[[other note]]`) are NOT resolved; they appear as literal text in `user_notes`. Notes with valid YAML frontmatter use Convention A (BibTeX-style: `citekey:`, `authors:`, `year:`, `title:`); notes without frontmatter fall through to Convention B (filename stem as citation_key, body as user_notes). Notes with malformed YAML frontmatter are rejected as `invalid_field_format` rather than silently re-classified as Convention B.
+
+## Validating adapter output
+
+Before using the passport in any ARS workflow:
+
+```bash
+python scripts/check_literature_corpus_schema.py \
+    --passport /tmp/passport.yaml \
+    --rejection-log /tmp/rejection_log.yaml
+```
+
+## How to write your own adapter
+
+1. Read the adapter contract at [`academic-pipeline/references/adapters/overview.md`](../../academic-pipeline/references/adapters/overview.md).
+2. Read the JSON schemas:
+   - [`shared/contracts/passport/literature_corpus_entry.schema.json`](../../shared/contracts/passport/literature_corpus_entry.schema.json)
+   - [`shared/contracts/passport/rejection_log.schema.json`](../../shared/contracts/passport/rejection_log.schema.json)
+3. Copy one of `folder_scan.py` / `zotero.py` / `obsidian.py` as a starting point.
+4. Change the input-reading and field-mapping code for your source.
+5. Keep the output shape exactly (sorted `literature_corpus[]` by `citation_key`, always emit `rejection_log.yaml`, schema-clean entries only).
+6. Set `obtained_via: "other"` and `adapter_name: "<your-adapter-name>"` on every entry you produce.
+7. Reuse helpers in [`_common.py`](_common.py) where possible — `make_citation_key`, `ensure_unique_citekey`, `parse_csl_name`, `parse_semicolon_names`, `path_to_file_uri`, `write_passport`, `write_rejection_log`, `now_iso`. They encode the contract details so you do not have to re-derive them.
+8. Validate: `python scripts/check_literature_corpus_schema.py --passport <your passport>`.
+9. Write tests modeled on `scripts/adapters/tests/`. The conftest fixtures `clean_timestamps` and `load_yaml` make golden-output testing easy.
+
+## Privacy reminder
+
+`abstract` and `user_notes` can contain publisher-copyrighted material. Before sharing a passport that contains these fields publicly (e.g., in a public repo or over the web), make sure you have the right to publish that text. ARS does not enforce this — it only warns you in the schema description.
+
+## Tests
+
+```bash
+cd <repo_root>
+pytest scripts/adapters/tests/ -v
+```
+
+CI runs these on every push / PR that touches `scripts/adapters/**`, `shared/contracts/passport/**`, the adapter overview, or either of the two lint scripts.

package/vendor/ars/scripts/adapters/_common.py

@@ -0,0 +1,209 @@
+"""Shared helpers for ARS reference adapters.
+
+These helpers are used by folder_scan.py, zotero.py, and obsidian.py.
+They are also what a user-written adapter would copy and adapt.
+"""
+from __future__ import annotations
+import datetime
+import io
+import re
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+
+def path_to_file_uri(path: Path | str) -> str:
+    """Build an RFC 8089 ``file://`` URI from a filesystem path.
+
+    Uses ``Path.as_uri()`` after resolving so spaces and reserved
+    characters are percent-encoded (``Lee 2024 paper.pdf`` →
+    ``file:///.../Lee%202024%20paper.pdf``). Naive ``f"file://{path}"``
+    yields invalid URIs and is the codex-flagged P1 from T7.
+    """
+    return Path(path).resolve().as_uri()
+
+ADAPTER_SPEC_VERSION = "1.0.0"  # bump when overview.md contract changes
+
+_NON_ALNUM = re.compile(r"[^A-Za-z0-9]+")
+_TITLE_WORD = re.compile(r"[A-Za-z]+")
+_STOPWORDS = {
+    "a", "an", "the", "of", "in", "on", "at", "by", "for", "to", "with",
+    "and", "or", "but", "is", "are", "was", "were",
+}
+
+
+def sanitize_citation_key(raw: str) -> str:
+    """Strip non-alphanumeric characters. Does NOT lowercase."""
+    return _NON_ALNUM.sub("", raw)
+
+
+def _first_title_word_non_stopword(title_hint: str | None) -> str:
+    if not title_hint:
+        return ""
+    for word in _TITLE_WORD.findall(title_hint):
+        if word.lower() not in _STOPWORDS:
+            return word.lower()
+    return ""
+
+
+def _alpha_suffixes():
+    """Yield 'a', 'b', ..., 'z', 'aa', 'ab', ..., 'zz'."""
+    for c in "abcdefghijklmnopqrstuvwxyz":
+        yield c
+    for c1 in "abcdefghijklmnopqrstuvwxyz":
+        for c2 in "abcdefghijklmnopqrstuvwxyz":
+            yield c1 + c2
+
+
+def make_citation_key(
+    *, family: str, year: int, title_hint: str | None, existing: set[str]
+) -> str:
+    """Build a collision-safe citation key.
+
+    Pattern: {family_lower}{year}{first_title_word_or_empty}. If that is
+    already in `existing`, append `a`, `b`, `c`, ... (then aa, ab, ...).
+    The chosen key is added to `existing` by reference.
+
+    Falls back to "ref" when the sanitized base is empty (e.g., empty
+    family with year=0 and no title_hint).
+    """
+    base = f"{family.lower()}{year}{_first_title_word_non_stopword(title_hint)}"
+    base = sanitize_citation_key(base)
+    # citation_key schema requires ^[A-Za-z]... so a pure-digit base
+    # (e.g. family='' year=2024) would fail schema validation. Fall back.
+    if not base or not base[0].isalpha():
+        base = "ref"
+
+    if base not in existing:
+        existing.add(base)
+        return base
+
+    for suffix in _alpha_suffixes():
+        candidate = f"{base}{suffix}"
+        if candidate not in existing:
+            existing.add(candidate)
+            return candidate
+    raise RuntimeError("exhausted citation key suffix space")  # practically unreachable
+
+
+def ensure_unique_citekey(key: str, existing: set[str]) -> str:
+    """Disambiguate a pre-existing citation key against ``existing``.
+
+    Used by adapters whose source already supplies a citekey (zotero
+    Better BibTeX, obsidian frontmatter ``citekey:``). The returned
+    key is sanitized to satisfy the schema pattern
+    ``^[A-Za-z][A-Za-z0-9_:-]*$`` and is added to ``existing`` by
+    reference. Collisions get an alpha suffix (a, b, ..., z, aa, ...)
+    just like ``make_citation_key``.
+
+    Empty / leading-digit / leading-punct bases fall back to ``ref``.
+    """
+    cleaned_chars = []
+    for ch in key:
+        if ch.isalnum() or ch in "_:-":
+            cleaned_chars.append(ch)
+    base = "".join(cleaned_chars)
+    if not base or not base[0].isalpha():
+        # Prefix with 'ref' so the result satisfies the schema's
+        # leading-letter constraint while preserving the original
+        # numeric/punctuated tail when one exists.
+        base = f"ref{base}" if base else "ref"
+
+    if base not in existing:
+        existing.add(base)
+        return base
+
+    for suffix in _alpha_suffixes():
+        candidate = f"{base}{suffix}"
+        if candidate not in existing:
+            existing.add(candidate)
+            return candidate
+    raise RuntimeError("exhausted citation key suffix space")
+
+
+def parse_csl_name(raw: str) -> dict[str, str]:
+    """Parse a single name string into a CSL-JSON name object.
+
+    Rules:
+    - "{Institution Name}" -> {"literal": "Institution Name"}
+    - "Family, Given" -> {"family": "Family", "given": "Given"}
+    - "Family" (single token) -> {"family": "Family"}
+    """
+    raw = raw.strip()
+    if raw.startswith("{") and raw.endswith("}"):
+        return {"literal": raw[1:-1].strip()}
+    if "," in raw:
+        family, _, given = raw.partition(",")
+        return {"family": family.strip(), "given": given.strip()}
+    return {"family": raw}
+
+
+def parse_semicolon_names(raw: str) -> list[dict[str, str]]:
+    """Parse 'A, B; C, D' into a CSL name list. Empty segments are skipped."""
+    raw = raw.strip()
+    if not raw:
+        return []
+    return [parse_csl_name(part) for part in raw.split(";") if part.strip()]
+
+
+def dump_yaml_stable(data: Any) -> str:
+    """Dump YAML with sorted keys and deterministic formatting."""
+    buf = io.StringIO()
+    yaml.safe_dump(
+        data,
+        buf,
+        sort_keys=True,
+        default_flow_style=False,
+        allow_unicode=True,
+        width=1000,
+    )
+    return buf.getvalue()
+
+
+def now_iso() -> str:
+    """ISO 8601 in UTC with trailing 'Z' (RFC 3339 compatible)."""
+    return datetime.datetime.now(datetime.UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def write_passport(path: Path, entries: list[dict[str, Any]]) -> None:
+    """Write passport.yaml with literature_corpus[] sorted by citation_key."""
+    sorted_entries = sorted(entries, key=lambda e: e.get("citation_key", ""))
+    doc = {"literature_corpus": sorted_entries}
+    path.write_text(dump_yaml_stable(doc), encoding="utf-8")
+
+
+def write_rejection_log(
+    path: Path,
+    *,
+    adapter_name: str,
+    adapter_version: str,
+    rejected: list[dict[str, Any]],
+    input_source: str | None = None,
+    total_input: int | None = None,
+    total_accepted: int | None = None,
+) -> None:
+    """Write rejection_log.yaml with rejected[] sorted by source.
+
+    summary.total_rejected is always set from len(rejected). When the caller
+    knows the upstream input or accept count, pass total_input / total_accepted
+    so the summary tells the full story; otherwise those keys are omitted.
+    """
+    sorted_rejected = sorted(rejected, key=lambda r: r.get("source", ""))
+    summary: dict[str, int] = {"total_rejected": len(sorted_rejected)}
+    if total_input is not None:
+        summary["total_input"] = total_input
+    if total_accepted is not None:
+        summary["total_accepted"] = total_accepted
+
+    doc: dict[str, Any] = {
+        "adapter_name": adapter_name,
+        "adapter_version": adapter_version,
+        "generated_at": now_iso(),
+        "rejected": sorted_rejected,
+        "summary": summary,
+    }
+    if input_source is not None:
+        doc["input_source"] = input_source
+
+    path.write_text(dump_yaml_stable(doc), encoding="utf-8")

package/vendor/ars/scripts/adapters/examples/folder_scan/expected_passport.yaml

@@ -0,0 +1,25 @@
+literature_corpus:
+- adapter_name: folder_scan.py
+  adapter_version: 1.1.0
+  authors:
+  - family: Chen
+  citation_key: chen2024aiassessment
+  obtained_at: '1970-01-01T00:00:00Z'
+  obtained_via: folder-scan
+  source_pointer: 'file:///FIXTURE/STRIPPED'
+  title: Chen2024 AIAssessment
+  venue_type: unknown
+  venue_type_provenance: unknown
+  year: 2024
+- adapter_name: folder_scan.py
+  adapter_version: 1.1.0
+  authors:
+  - family: Wang
+  citation_key: wang2023formative
+  obtained_at: '1970-01-01T00:00:00Z'
+  obtained_via: folder-scan
+  source_pointer: 'file:///FIXTURE/STRIPPED'
+  title: Wang 2023 formative feedback
+  venue_type: unknown
+  venue_type_provenance: unknown
+  year: 2023

package/vendor/ars/scripts/adapters/examples/folder_scan/expected_rejection_log.yaml

@@ -0,0 +1,18 @@
+adapter_name: folder_scan.py
+adapter_version: 1.1.0
+generated_at: '1970-01-01T00:00:00Z'
+input_source: '/FIXTURE/STRIPPED'
+rejected:
+- missing_fields:
+  - authors
+  - year
+  raw: paper1.pdf
+  reason: authors_unparseable
+  source: paper1.pdf
+- missing_fields:
+  - authors
+  raw: 中文檔名_2024.pdf
+  reason: authors_unparseable
+  source: 中文檔名_2024.pdf
+summary:
+  total_rejected: 2

package/vendor/ars/scripts/adapters/examples/obsidian/expected_passport.yaml

@@ -0,0 +1,40 @@
+literature_corpus:
+- adapter_name: obsidian.py
+  adapter_version: 1.1.0
+  authors:
+  - family: Chen
+    given: Cindy
+  citation_key: chen2024ai
+  doi: 10.1234/xyz
+  obtained_at: '1970-01-01T00:00:00Z'
+  obtained_via: obsidian-vault
+  source_pointer: obsidian://open?vault=vault&file=chen2024ai
+  tags:
+  - assessment
+  - ai
+  title: AI assessment in higher education
+  user_notes: 'Body note content. This may contain copyrighted excerpts.
+
+    '
+  venue: Assessment & Evaluation
+  venue_type: unknown
+  venue_type_provenance: unknown
+  year: 2024
+- adapter_name: obsidian.py
+  adapter_version: 1.1.0
+  authors:
+  - family: Wang
+    given: J.
+  - family: Lin
+    given: S.
+  citation_key: wang2023formative
+  obtained_at: '1970-01-01T00:00:00Z'
+  obtained_via: obsidian-vault
+  source_pointer: obsidian://open?vault=vault&file=wang2023formative
+  title: Formative feedback in classrooms
+  user_notes: 'Body content here.
+
+    '
+  venue_type: unknown
+  venue_type_provenance: unknown
+  year: 2023

package/vendor/ars/scripts/adapters/examples/obsidian/expected_rejection_log.yaml

@@ -0,0 +1,13 @@
+adapter_name: obsidian.py
+adapter_version: 1.1.0
+generated_at: '1970-01-01T00:00:00Z'
+rejected:
+- missing_fields:
+  - title
+  - authors
+  - year
+  raw: invalid.md
+  reason: authors_unparseable
+  source: invalid.md
+summary:
+  total_rejected: 1

package/vendor/ars/scripts/adapters/examples/obsidian/input_fixture/vault/.obsidian/app.json

@@ -0,0 +1 @@
+{"promptDelete": false}

package/vendor/ars/scripts/adapters/examples/obsidian/input_fixture/vault/_templates/tmpl.md

@@ -0,0 +1,7 @@
+---
+citekey: tmpl2024
+title: Template — should be skipped
+authors:
+  - family: Ignore
+year: 2024
+---

package/vendor/ars/scripts/adapters/examples/obsidian/input_fixture/vault/chen2024ai.md

@@ -0,0 +1,13 @@
+---
+citekey: chen2024ai
+title: AI assessment in higher education
+authors:
+  - family: Chen
+    given: Cindy
+year: 2024
+venue: Assessment & Evaluation
+doi: 10.1234/xyz
+tags: [assessment, ai]
+---
+
+Body note content. This may contain copyrighted excerpts.

package/vendor/ars/scripts/adapters/examples/obsidian/input_fixture/vault/invalid.md

@@ -0,0 +1 @@
+This file has no frontmatter and no parseable H1. It should be rejected.

package/vendor/ars/scripts/adapters/examples/obsidian/input_fixture/vault/wang2023formative.md

@@ -0,0 +1,10 @@
+---
+source: https://doi.org/10.5678/abc
+---
+
+# Formative feedback in classrooms
+
+**Authors**: Wang, J.; Lin, S.
+**Year**: 2023
+
+Body content here.

package/vendor/ars/scripts/adapters/examples/zotero/expected_passport.yaml

@@ -0,0 +1,34 @@
+literature_corpus:
+- abstract: An abstract that is publisher-copyrighted.
+  adapter_name: zotero.py
+  adapter_version: 1.1.0
+  authors:
+  - family: Chen
+    given: Cindy
+  - family: Wang
+    given: Jing
+  citation_key: chen2024ai
+  doi: 10.1234/xyz
+  obtained_at: '1970-01-01T00:00:00Z'
+  obtained_via: zotero-bbt-export
+  source_pointer: zotero://select/items/0_ABCD1234
+  tags:
+  - assessment
+  - ai
+  title: AI assessment in higher education
+  venue: Assessment & Evaluation
+  venue_type: journal-article
+  venue_type_provenance: adapter_declared
+  year: 2024
+- adapter_name: zotero.py
+  adapter_version: 1.1.0
+  authors:
+  - literal: World Health Organization
+  citation_key: who2024report
+  obtained_at: '1970-01-01T00:00:00Z'
+  obtained_via: zotero-bbt-export
+  source_pointer: zotero://select/items/0_EFGH5678
+  title: World higher-education report
+  venue_type: report
+  venue_type_provenance: adapter_declared
+  year: 2024

package/vendor/ars/scripts/adapters/examples/zotero/expected_rejection_log.yaml

@@ -0,0 +1,34 @@
+adapter_name: zotero.py
+adapter_version: 1.1.0
+generated_at: '1970-01-01T00:00:00Z'
+rejected:
+- missing_fields:
+  - year
+  raw:
+    citationKey: forthcoming2024
+    creators:
+    - creatorType: author
+      firstName: X
+      lastName: Y
+    date: forthcoming
+    itemID: IJKL9012
+    itemType: journalArticle
+    title: To appear
+  reason: year_unparseable
+  source: IJKL9012
+- missing_fields:
+  - authors
+  raw:
+    citationKey: editor2023
+    creators:
+    - creatorType: editor
+      firstName: E
+      lastName: Z
+    date: '2023'
+    itemID: MNOP3456
+    itemType: book
+    title: Edited volume
+  reason: authors_unparseable
+  source: MNOP3456
+summary:
+  total_rejected: 2

package/vendor/ars/scripts/adapters/examples/zotero/input_fixture/export.json

@@ -0,0 +1,43 @@
+[
+  {
+    "citationKey": "chen2024ai",
+    "itemType": "journalArticle",
+    "title": "AI assessment in higher education",
+    "creators": [
+      {"creatorType": "author", "firstName": "Cindy", "lastName": "Chen"},
+      {"creatorType": "author", "firstName": "Jing", "lastName": "Wang"}
+    ],
+    "date": "2024",
+    "publicationTitle": "Assessment & Evaluation",
+    "DOI": "10.1234/xyz",
+    "tags": [{"tag": "assessment"}, {"tag": "ai"}],
+    "itemID": "ABCD1234",
+    "abstractNote": "An abstract that is publisher-copyrighted."
+  },
+  {
+    "citationKey": "who2024report",
+    "itemType": "report",
+    "title": "World higher-education report",
+    "creators": [
+      {"creatorType": "author", "name": "World Health Organization"}
+    ],
+    "date": "2024-03",
+    "itemID": "EFGH5678"
+  },
+  {
+    "citationKey": "forthcoming2024",
+    "itemType": "journalArticle",
+    "title": "To appear",
+    "creators": [{"creatorType": "author", "firstName": "X", "lastName": "Y"}],
+    "date": "forthcoming",
+    "itemID": "IJKL9012"
+  },
+  {
+    "citationKey": "editor2023",
+    "itemType": "book",
+    "title": "Edited volume",
+    "creators": [{"creatorType": "editor", "firstName": "E", "lastName": "Z"}],
+    "date": "2023",
+    "itemID": "MNOP3456"
+  }
+]