okf-loremaster 0.1.0__tar.gz

okf_loremaster-0.1.0/.env.example

@@ -0,0 +1,158 @@
+# OKF Loremaster — copy to .env and fill in. Never put real values in this file.
+# Every variable is read through okf_loremaster.config; nothing is read from os.environ directly.
+
+# ---------------------------------------------------------------------------
+# LLM routing (required)
+# ---------------------------------------------------------------------------
+# Model strings are passed verbatim to LiteLLM, so any LiteLLM-supported
+# provider works. Three roles; bind them to whatever tier you like.
+#   FAST       cheapest and quickest; screening, one abstract at a time
+#   BALANCED   mid-priced; query planning, curation, and extraction
+#   REASONING  most capable and most expensive; the charter, and nothing else
+#
+# Extraction sits on BALANCED because it is the only node that makes a call per
+# paper — two hundred of them against every other node's handful — so whatever it
+# is bound to sets the price of a run. What guards the output there is code, not
+# model tier: every number is checked against the source afterward and every quote
+# is sliced out of it. Bind REASONING to something cheaper and you will barely
+# notice; bind BALANCED to something expensive and you will.
+OKF_LOREMASTER_MODEL_FAST=claude-haiku-4-5
+OKF_LOREMASTER_MODEL_BALANCED=claude-sonnet-5
+OKF_LOREMASTER_MODEL_REASONING=claude-opus-5
+
+ANTHROPIC_API_KEY=
+# Optional: gateway or Azure-style base URL, e.g.
+# ANTHROPIC_BASE_URL=https://<resource>.services.ai.azure.com/anthropic/
+ANTHROPIC_BASE_URL=
+
+# ---------------------------------------------------------------------------
+# Cost accounting
+# ---------------------------------------------------------------------------
+# LiteLLM's completion_cost() returns 0.0 for models it does not know, which is
+# indistinguishable from a free call. Models reached through a gateway or an
+# Azure deployment name are commonly absent from its price map. Set these to get
+# real USD figures; leave them unset and unpriced calls are reported as tokens
+# with "cost unavailable" — never $0.00.
+# Units: USD per 1M tokens.
+# OKF_LOREMASTER_PRICE_FAST_IN=
+# OKF_LOREMASTER_PRICE_FAST_OUT=
+# OKF_LOREMASTER_PRICE_BALANCED_IN=
+# OKF_LOREMASTER_PRICE_BALANCED_OUT=
+# OKF_LOREMASTER_PRICE_REASONING_IN=
+# OKF_LOREMASTER_PRICE_REASONING_OUT=
+
+# Soft budget in USD. The run warns and pauses at this threshold; it does not abort.
+# OKF_LOREMASTER_MAX_USD=5.00
+
+# ---------------------------------------------------------------------------
+# Model concurrency and retries
+# ---------------------------------------------------------------------------
+# How many calls per tier may be in flight at once. Screening submits every pooled
+# paper at once and these are what meter it, so FAST is the one that matters. The
+# binding limit is usually tokens per minute rather than requests: if a run reports
+# screening calls failing with RateLimitError, lower FAST before anything else.
+# OKF_LOREMASTER_CONCURRENCY_FAST=4
+# BALANCED is what sets a run's wall-clock: extraction is one call per kept paper, so a
+# 200-paper bundle makes 200 of them. At 2 that is hours. Lower it only if extraction
+# starts reporting RateLimitError.
+# OKF_LOREMASTER_CONCURRENCY_BALANCED=3
+# OKF_LOREMASTER_CONCURRENCY_REASONING=3
+
+# Attempts per call, not retries on top of the first — warnings count to one less.
+# A rate limit clears on a 60-second window, so this has to be enough to outlast one.
+# OKF_LOREMASTER_MAX_RETRIES=6
+
+# Seconds before a single model call is abandoned. An extraction reads 6,000 tokens of
+# source and writes several thousand back, which is minutes rather than seconds; set too
+# low, the call times out on its own success and the paper is lost.
+# OKF_LOREMASTER_REQUEST_TIMEOUT=300
+
+# ---------------------------------------------------------------------------
+# NCBI E-utilities
+# ---------------------------------------------------------------------------
+# Optional but strongly recommended: raises the rate limit from 3 to 10 req/s.
+# Free from https://account.ncbi.nlm.nih.gov/settings/
+# We run at 2.5 or 8 req/s respectively — under the ceiling on purpose, because
+# the limit is enforced per IP and a shared address may already be carrying
+# traffic we cannot see. E-utilities, BioC and PubTator share one budget.
+OKF_LOREMASTER_NCBI_API_KEY=
+# Required. NCBI asks for a contact address on every request and throttles
+# traffic that omits it. A build refuses to start without it.
+OKF_LOREMASTER_NCBI_EMAIL=
+OKF_LOREMASTER_NCBI_TOOL=okf-loremaster
+
+# ---------------------------------------------------------------------------
+# HTTP
+# ---------------------------------------------------------------------------
+# OKF_LOREMASTER_HTTP_TIMEOUT=30
+# OKF_LOREMASTER_HTTP_MAX_RETRIES=4
+# Responses are cached on disk, keyed by the request with credentials stripped,
+# so rotating an API key does not orphan the cache. Bibliographic records are
+# effectively immutable, hence the long default.
+# OKF_LOREMASTER_HTTP_CACHE_ENABLED=true
+# OKF_LOREMASTER_HTTP_CACHE_TTL_DAYS=30
+#
+# A CA bundle to verify TLS against, instead of the default trust store. Needed on
+# a network whose proxy terminates TLS and presents its own certificate: the run
+# then reports a certificate failure against a host that is perfectly healthy, and
+# it can hit some hosts and not others depending on what the proxy allowlists.
+# Ask your IT group for the organization's root CA in PEM form. There is
+# deliberately no option to skip verification — the provenance a bundle claims
+# rests on the bytes having come from who they say they came from.
+# OKF_LOREMASTER_CA_BUNDLE=/path/to/corporate-ca.pem
+
+# ---------------------------------------------------------------------------
+# Embeddings (only used by the optional vector index)
+# ---------------------------------------------------------------------------
+# Must be a locally-runnable model: downstream consumers reject remote embedders
+# on attach. Pinned by revision so a rebuild reproduces the same vectors.
+# OKF_LOREMASTER_EMBED_MODEL=pritamdeka/S-PubMedBert-MS-MARCO
+# OKF_LOREMASTER_EMBED_REVISION=
+
+# Shared Hugging Face cache, so the embedding model downloads once per machine
+# rather than once per environment. Use an absolute path — a literal ~ is not
+# expanded here. Keep it OUTSIDE OneDrive: the hub cache symlinks snapshots/
+# into blobs/, which sync clients mangle.
+HF_HOME=/Users/<you>/.cache/huggingface
+
+# ---------------------------------------------------------------------------
+# Review
+# ---------------------------------------------------------------------------
+# Who a `--review` sign-off is attributed to. Recorded in every document as
+# verified: [{by: "human:<id>", ...}], which is what lifts the bundle above the
+# unverified trust tier — so it has to name someone who can be asked about it.
+# Unset falls back to the OS login name; set it when that is a service account
+# or a shared box.
+# OKF_LOREMASTER_REVIEWER_ID=
+
+# ---------------------------------------------------------------------------
+# Paths
+# ---------------------------------------------------------------------------
+# HTTP response cache and run checkpoints. Defaults to a platform cache dir.
+# OKF_LOREMASTER_CACHE_DIR=
+# Where runs are written. Defaults to ./bundles relative to the working dir.
+# Every run lands here: `-o <name>` takes a name, not a path, and it is resolved
+# against this. An absolute -o overrides it.
+# OKF_LOREMASTER_OUTPUT_DIR=
+#
+# How many past runs keep their checkpoints. A build writes 100 to 350 MB of them
+# — the whole state is saved once per node, and by the later ones it holds full
+# texts and extractions — so without this the cache directory grows forever. Each
+# build drops all but this many, newest first; 0 keeps everything. Only affects
+# what `--resume` can still pick up. Bundles are never touched.
+# OKF_LOREMASTER_CHECKPOINT_KEEP_RUNS=5
+#
+# Ceilings in MB on the three things that accumulate. Applied at both ends of a
+# build, oldest entries first, so between builds each store is under its ceiling and
+# only a run in flight is over. Nothing is reclaimed unless you build. 0 turns one
+# off. `okf-loremaster runs` prints each size against its cap.
+#
+# Checkpoints are scratch, and the count above is meant to be what binds: five runs
+# measure around 900 MB, so this is set clear of that rather than quietly cutting how
+# many are resumable. The other two are not scratch — they are keyed by the request
+# and shared by every bundle, which is what makes a rebuild fast and re-reading a
+# paper free — so they sit where an ordinary year of use will not reach them. Raise
+# the last one before the others if you rerun often.
+# OKF_LOREMASTER_CHECKPOINT_MAX_MB=2048
+# OKF_LOREMASTER_HTTP_CACHE_MAX_MB=1024
+# OKF_LOREMASTER_EXTRACTION_CACHE_MAX_MB=512

okf_loremaster-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,86 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: pyproject.toml
+
+      # `[all]` rather than the base install: the Chroma and Textual tests
+      # `importorskip` their dependency, so a lean install would report them as passing
+      # when nothing ran. The torch download is the price of actually covering them.
+      - name: Install
+        run: pip install -e ".[all,dev]"
+
+      # `ruff check` only. The project does not use `ruff format` and is not formatted to
+      # it — adding the check here would fail on 48 files that are deliberately hand-set.
+      - name: Lint
+        run: ruff check src/ tests/
+
+      # Only on the version mypy is configured for. `python_version = "3.11"` in
+      # `pyproject.toml` is the point of the check — the floor is what has to typecheck —
+      # but under 3.12 mypy reads that interpreter's numpy stubs, which use `type`
+      # statements, and rejects 3.12 syntax it was told to treat as 3.11.
+      - name: Types
+        if: matrix.python-version == '3.11'
+        run: mypy src/
+
+      # No network reachable from the suite by design — conftest blocks the httpx
+      # transports outright, so a test that starts calling out fails rather than
+      # depending on NCBI being up.
+      - name: Tests
+        run: pytest -q
+
+  package:
+    name: Build and inspect the distributions
+    runs-on: ubuntu-latest
+    steps:
+      # The whole history and all tags: `check_dist.py` compares the sdist against
+      # `git ls-files`, which needs a real repository rather than an export.
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install build tooling
+        run: pip install build twine
+
+      - name: Build
+        run: python -m build
+
+      # Fails if the sdist contains a path git does not track, if any `.env` appears,
+      # or if the packaged `.env` template is missing from the wheel.
+      - name: Nothing untracked ships
+        run: python scripts/check_dist.py dist/
+
+      - name: Metadata renders on PyPI
+        run: twine check --strict dist/*
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: distributions
+          path: dist/

okf_loremaster-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,152 @@
+# Publishing is driven by the tag, and the tag alone decides where it lands:
+#
+#   v0.1.0rc1, v0.1.0a2, v0.1.0b1   → TestPyPI      (rehearsal; the page can be looked at)
+#   v0.1.0                          → PyPI          (permanent; a version is never reusable)
+#
+# Uploads use PyPI Trusted Publishing (OpenID Connect), so there is no API token in the
+# repository to leak or rotate. Each index needs a publisher registered once, against this
+# repository, this workflow filename, and the environment named below.
+name: Release
+
+on:
+  push:
+    tags: ["v*"]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    outputs:
+      prerelease: ${{ steps.classify.outputs.prerelease }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install build tooling
+        run: pip install build twine
+
+      # The version is read from `src/okf_loremaster/__init__.py`, so a tag that disagrees
+      # with it would publish something named differently from what the CLI reports.
+      - name: Tag matches the package version
+        if: startsWith(github.ref, 'refs/tags/')
+        run: |
+          tag="${GITHUB_REF_NAME#v}"
+          pkg="$(python -c 'import re,pathlib; print(re.search(r"__version__ = \"([^\"]+)\"", pathlib.Path("src/okf_loremaster/__init__.py").read_text()).group(1))')"
+          if [ "$tag" != "$pkg" ]; then
+            echo "tag $GITHUB_REF_NAME says $tag, __version__ says $pkg — retag or bump" >&2
+            exit 1
+          fi
+          echo "both say $pkg"
+
+      - name: Classify the tag
+        id: classify
+        run: |
+          if printf '%s' "${GITHUB_REF_NAME}" | grep -qE '(a|b|rc)[0-9]+$'; then
+            echo "prerelease=true" >> "$GITHUB_OUTPUT"
+            echo "${GITHUB_REF_NAME} is a pre-release — TestPyPI"
+          else
+            echo "prerelease=false" >> "$GITHUB_OUTPUT"
+            echo "${GITHUB_REF_NAME} is a final release — PyPI"
+          fi
+
+      - name: Build
+        run: python -m build
+
+      - name: Nothing untracked ships
+        run: python scripts/check_dist.py dist/
+
+      - name: Metadata renders on PyPI
+        run: twine check --strict dist/*
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: distributions
+          path: dist/
+
+  # Installs the built wheel in a clean environment with no checkout on the path, which is
+  # the only way to catch a file that the repository has and the wheel does not. `init` is
+  # the command that fails that way: it needs the packaged .env template.
+  smoke-test:
+    needs: build
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - uses: actions/download-artifact@v4
+        with:
+          name: distributions
+          path: dist/
+
+      - name: Install the wheel alone
+        run: pip install dist/*.whl
+
+      - name: It runs, and writes a template from somewhere other than a checkout
+        run: |
+          mkdir -p /tmp/elsewhere && cd /tmp/elsewhere
+          okf-loremaster --version
+          okf-loremaster --help > /dev/null
+          okf-loremaster init || true
+          test -s .env || { echo "init wrote no .env — the packaged template is missing" >&2; exit 1; }
+          grep -q '^ANTHROPIC_API_KEY=$' .env || { echo ".env template is not blank" >&2; exit 1; }
+          echo "init wrote a blank .env from the packaged template"
+
+  testpypi:
+    needs: [build, smoke-test]
+    if: needs.build.outputs.prerelease == 'true'
+    runs-on: ubuntu-latest
+    environment:
+      name: testpypi
+      url: https://test.pypi.org/p/okf-loremaster
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: distributions
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: https://test.pypi.org/legacy/
+
+  pypi:
+    needs: [build, smoke-test]
+    if: needs.build.outputs.prerelease == 'false' && startsWith(github.ref, 'refs/tags/')
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/okf-loremaster
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: distributions
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1
+
+  # Last, so a GitHub release only ever describes something that actually published.
+  github-release:
+    needs: pypi
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: distributions
+          path: dist/
+      - uses: softprops/action-gh-release@v2
+        with:
+          files: dist/*
+          generate_release_notes: true

okf_loremaster-0.1.0/.gitignore

@@ -0,0 +1,53 @@
+# Secrets. `.env.local` and `.env.production` matched neither of the first two
+# patterns — `.env` is exact and `*.env` needs something before the dot — so the third
+# is the one that closes it. The exception has to come last to survive them all.
+.env
+*.env
+.env.*
+!.env.example
+
+__pycache__/
+*.py[cod]
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+*.egg-info/
+build/
+dist/
+.coverage
+htmlcov/
+
+# Build outputs: bundles are regenerable and large.
+bundles/
+*.chroma/
+.okf-loremaster-cache/
+
+# Run artifacts. Checkpoints normally live under the cache directory, outside the
+# repo — these catch a cache dir pointed back inside it.
+*.sqlite
+*.sqlite-journal
+
+# A charter is one run's input, not the package's. Anchored to the root so a
+# fixture or example charter under tests/ still ships.
+/charter.yaml
+/*.charter.yaml
+
+# Internal working notes: the agent operating manual and the datestamped build log.
+# Kept local — they are about how this was built, not about how it is used.
+/CLAUDE.md
+/Build_Progress.md
+
+# Agent tooling state: local permissions, session files, transcripts. Machine- and
+# operator-specific, and none of it describes the package.
+.claude/
+
+# A virtualenv inside the repo is against policy — the conda env is kept outside so it
+# does not bloat cloud sync — but a stray one committed is thousands of files, so it is
+# ignored rather than trusted not to appear.
+.venv/
+venv/
+
+.DS_Store
+.vscode/
+.idea/
+.ipynb_checkpoints/

okf_loremaster-0.1.0/CHANGELOG.md

@@ -0,0 +1,44 @@
+# Changelog
+
+Notable changes to OKF Loremaster. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
+versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Version 0.x means the OKF bundle layout and the CLI may still change between minor releases.
+The bundle contract downstream reads is the part to watch — it is called out here when it moves.
+
+## [Unreleased]
+
+## [0.1.0] — 2026-08-14
+
+First public release.
+
+### Added
+
+- Build a task-scoped biomedical literature bundle from PubMed/PMC in Open Knowledge Format
+  v0.2, from a plain-language research question: `okf-loremaster build "..."`.
+- A thirteen-node graph — `charter → search → dedupe → rank → screen → curate → fulltext →
+  extract → reconcile → review → emit_okf → validate → index_vectors`. Five agents make the
+  judgment calls; HTTP, dedup, ranking, MMR, license logic, validation and indexing are code.
+- Per-paper markdown carrying predictor rows with effect sizes, operationalization, timing,
+  evidence type, null findings and vocabulary hints, plus a `predictors.md` index that points
+  into them and a `search.md` that reproduces the search.
+- Deterministic numeric verification: an effect the source text does not contain becomes
+  `effect=None` with a downgraded confidence and a logged warning, and the run continues.
+- A computed `strength` per row from study design, adjustment and sample-size scale — never
+  asked of the model.
+- Optional Chroma vector index over the finished bundle (`[vectors]`), built by walking what
+  was emitted rather than by a second extraction pass.
+- Resume from a checkpoint (`--resume`), a `runs` listing, a Textual TUI (`[tui]`), JSONL
+  event output, and cost reporting that says "cost unavailable" rather than `$0.00` for a
+  model LiteLLM cannot price.
+- `okf-loremaster init` writes an annotated `.env` and reports what is still unset.
+
+### Notes
+
+- Python 3.11 and 3.12.
+- NCBI access is E-utilities, BioC, PubTator and iCite only; no page is ever scraped.
+- Apache-2.0 covers this code, not the bundles it builds. Each emitted document records the
+  `license` its publisher reported, and `export_safe` says whether it may leave.
+
+[Unreleased]: https://github.com/PFreda-Lab/okf-loremaster/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/PFreda-Lab/okf-loremaster/releases/tag/v0.1.0