baseqdb 0.5.0__tar.gz

baseqdb-0.5.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 baSeq ApS
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

baseqdb-0.5.0/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.5
+Name: baseqdb
+Version: 0.5.0
+Summary: Your baSeq datasets, analyses and results from Python, with lamindb's names
+Project-URL: Homepage, https://baseq.bio
+Project-URL: Documentation, https://baseq.bio/docs/python-and-r.html
+Author-email: baSeq ApS <info@baseq.dk>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: baseq,bioinformatics,lineage,provenance,rna-seq,single-cell
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.10
+Requires-Dist: httpx>=0.24
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.5; extra == 'pandas'
+Description-Content-Type: text/markdown
+
+# baseqdb
+
+Your baSeq datasets, the analyses built on them, what they found and where a result came from, from Python.
+
+```python
+import baseqdb as bq
+bq.login()                                      # reads BASEQ_API_KEY
+
+bq.Dataset.filter(kind="bulk").to_dataframe()   # yours, newest versions
+bq.Analysis.filter(kind="bulk").to_dataframe()  # the analyses you can open, newest first
+ds = bq.Dataset.get("kTIqMTTm89P2gbNU0000")     # one version, by its uid
+folder = ds.cache()                             # its files, downloaded once
+sheet = ds.sample_sheet                         # a DataFrame
+
+analysis = ds.analyses[0]
+analysis.steps.to_dataframe()                   # each step, its settings, when it ran
+analysis.findings.to_dataframe()                # what each step found
+table = analysis.tables["dge_condition_dexamethasone_vs_untreated.csv"].load()
+
+record = bq.Record.get("Z3HMY4V1wC6goA380000")  # a file a step wrote
+record.run.inputs                               # the records it read
+ds.run.initiated_by                             # the run a version added from code was added in
+```
+
+With a key that may add data ("Read and add data"), add a dataset. With the name and kind of one you have, it is its
+next version, which starts what that dataset's "When a new version is added" says:
+
+```python
+new = bq.Dataset(["counts_filtered.tsv"], name="Airway demo", kind="bulk", sample_sheet="samples.csv").save()
+new.version                                     # 2: the next version
+new.notes                                       # what else to know, such as what the new version started
+```
+
+## Track your code
+
+One line at the top makes a script one run on baSeq, as lamindb's `ln.track()`: its code (kept as a version of the
+script), its command line and environment, the files it downloads from baSeq (what it read), the datasets it adds and
+the files it saves, and what it found when it says. Reads are counted, not listed one by one; everything a run adds is
+kept. Its page on baSeq is under Scripts, Runs outside baSeq; what it found is in your lab's findings too, marked From
+code, never taken for a step's checked finding.
+
+```python
+bq.track()                                      # this script, as one run
+counts = ds.files["counts.tsv"].load()          # noted as what the run read
+bq.Record("dge_dex.csv").save()                 # a file of the run
+bq.finish(finding="Dexamethasone raised ZBTB16 and FKBP5 the most.",
+          genes=["ZBTB16+", "FKBP5+"])          # what it found; or let the script end: an exception ends it as errored
+```
+
+## With an AI agent
+
+An agent such as Claude Code can use baseqdb with your key. Ask it to begin each task with `baseqdb track` and end it
+with `baseqdb finish`: everything it runs in that folder in between is one run, however many commands it takes.
+
+```sh
+baseqdb track "Dexamethasone response in Airway"
+python filter_counts.py --min-reads 10          # joins the run
+python dge_dex.py --padj 0.01                   # joins the run
+baseqdb finish --finding "ZBTB16 and FKBP5 rose the most." --gene ZBTB16+ --gene FKBP5+
+```
+
+If you are an agent: begin the task with `baseqdb track "<what the task is>"` in its folder, call `bq.track()` at the
+top of each script you write, save results you keep with `bq.Record(path).save()`, and end with `baseqdb finish`,
+saying in one short paragraph what the task found (`--finding`, with `--gene` for each gene; `--errored` when the task
+failed).
+
+Data you fetch with a key leaves baSeq. Our Data Processing Agreement does not cover what an AI agent or another
+service then does with it.
+
+Make an API key on your baSeq Account page, under API keys. Install pandas too for DataFrames:
+`pip install "baseqdb[pandas]"`.
+
+The whole guide, with R: https://baseq.bio/docs/python-and-r.html

baseqdb-0.5.0/README.md

@@ -0,0 +1,73 @@
+# baseqdb
+
+Your baSeq datasets, the analyses built on them, what they found and where a result came from, from Python.
+
+```python
+import baseqdb as bq
+bq.login()                                      # reads BASEQ_API_KEY
+
+bq.Dataset.filter(kind="bulk").to_dataframe()   # yours, newest versions
+bq.Analysis.filter(kind="bulk").to_dataframe()  # the analyses you can open, newest first
+ds = bq.Dataset.get("kTIqMTTm89P2gbNU0000")     # one version, by its uid
+folder = ds.cache()                             # its files, downloaded once
+sheet = ds.sample_sheet                         # a DataFrame
+
+analysis = ds.analyses[0]
+analysis.steps.to_dataframe()                   # each step, its settings, when it ran
+analysis.findings.to_dataframe()                # what each step found
+table = analysis.tables["dge_condition_dexamethasone_vs_untreated.csv"].load()
+
+record = bq.Record.get("Z3HMY4V1wC6goA380000")  # a file a step wrote
+record.run.inputs                               # the records it read
+ds.run.initiated_by                             # the run a version added from code was added in
+```
+
+With a key that may add data ("Read and add data"), add a dataset. With the name and kind of one you have, it is its
+next version, which starts what that dataset's "When a new version is added" says:
+
+```python
+new = bq.Dataset(["counts_filtered.tsv"], name="Airway demo", kind="bulk", sample_sheet="samples.csv").save()
+new.version                                     # 2: the next version
+new.notes                                       # what else to know, such as what the new version started
+```
+
+## Track your code
+
+One line at the top makes a script one run on baSeq, as lamindb's `ln.track()`: its code (kept as a version of the
+script), its command line and environment, the files it downloads from baSeq (what it read), the datasets it adds and
+the files it saves, and what it found when it says. Reads are counted, not listed one by one; everything a run adds is
+kept. Its page on baSeq is under Scripts, Runs outside baSeq; what it found is in your lab's findings too, marked From
+code, never taken for a step's checked finding.
+
+```python
+bq.track()                                      # this script, as one run
+counts = ds.files["counts.tsv"].load()          # noted as what the run read
+bq.Record("dge_dex.csv").save()                 # a file of the run
+bq.finish(finding="Dexamethasone raised ZBTB16 and FKBP5 the most.",
+          genes=["ZBTB16+", "FKBP5+"])          # what it found; or let the script end: an exception ends it as errored
+```
+
+## With an AI agent
+
+An agent such as Claude Code can use baseqdb with your key. Ask it to begin each task with `baseqdb track` and end it
+with `baseqdb finish`: everything it runs in that folder in between is one run, however many commands it takes.
+
+```sh
+baseqdb track "Dexamethasone response in Airway"
+python filter_counts.py --min-reads 10          # joins the run
+python dge_dex.py --padj 0.01                   # joins the run
+baseqdb finish --finding "ZBTB16 and FKBP5 rose the most." --gene ZBTB16+ --gene FKBP5+
+```
+
+If you are an agent: begin the task with `baseqdb track "<what the task is>"` in its folder, call `bq.track()` at the
+top of each script you write, save results you keep with `bq.Record(path).save()`, and end with `baseqdb finish`,
+saying in one short paragraph what the task found (`--finding`, with `--gene` for each gene; `--errored` when the task
+failed).
+
+Data you fetch with a key leaves baSeq. Our Data Processing Agreement does not cover what an AI agent or another
+service then does with it.
+
+Make an API key on your baSeq Account page, under API keys. Install pandas too for DataFrames:
+`pip install "baseqdb[pandas]"`.
+
+The whole guide, with R: https://baseq.bio/docs/python-and-r.html

baseqdb-0.5.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[project]
+name = "baseqdb"
+version = "0.5.0"
+description = "Your baSeq datasets, analyses and results from Python, with lamindb's names"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "baSeq ApS", email = "info@baseq.dk" }]
+keywords = ["bioinformatics", "rna-seq", "single-cell", "lineage", "provenance", "baseq"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Bio-Informatics",
+]
+dependencies = ["httpx>=0.24"]
+
+[project.scripts]
+baseqdb = "baseqdb._cli:main"
+
+[project.optional-dependencies]
+pandas = ["pandas>=1.5"]
+
+[project.urls]
+Homepage = "https://baseq.bio"
+Documentation = "https://baseq.bio/docs/python-and-r.html"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/baseqdb"]

baseqdb-0.5.0/src/baseqdb/__init__.py

@@ -0,0 +1,21 @@
+"""baseqdb: your baSeq datasets, the analyses built on them, what they found and where a result came from, from
+Python, with lamindb's names; and, with a key that may add data, new datasets and versions.
+
+    import baseqdb as bq
+    bq.login()                                      # reads BASEQ_API_KEY
+    bq.Dataset.filter(kind="bulk").to_dataframe()   # yours, newest versions
+    ds = bq.Dataset.get("kTIqMTTm89P2gbNU0000")     # one version, by its uid
+    folder = ds.cache()                             # its files, downloaded once
+    bq.Dataset(["counts.tsv"], name=ds.name, kind="bulk").save()   # its next version
+
+    bq.track()                                      # this script as one run: what it reads, adds and saves
+    bq.Record("dge.csv").save()                     # a file of the run
+    bq.finish()                                     # or let the script end
+"""
+from baseqdb._client import VERSION as __version__, BaseqError, login, me
+from baseqdb._records import (Analysis, Dataset, File, Files, Finding, GeneHit, Record, Results, Run, Step, Table,
+                              Tables)
+from baseqdb._track import finish, track
+
+__all__ = ["login", "me", "BaseqError", "Dataset", "File", "Files", "Analysis", "Step", "Finding", "GeneHit", "Table",
+           "Tables", "Record", "Run", "Results", "track", "finish", "__version__"]

baseqdb-0.5.0/src/baseqdb/_cli.py

@@ -0,0 +1,79 @@
+"""The ``baseqdb`` command, for an AI agent's task (or yours) as one run on baSeq:
+
+    baseqdb track "Dexamethasone response in Airway"     # begins the task's run; this folder's commands join it
+    python filter_counts.py                              # its requests name the task; bq.track() inside joins it
+    baseqdb finish --finding "ZBTB16 rose the most."     # ends it, with what it found (--gene ZBTB16+, --errored)
+
+The task's uid is kept in .baseqdb/run in this folder (BASEQ_RUN overrides it), which baseqdb in this folder and the
+folders below it reads."""
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from baseqdb import _client as _c
+from baseqdb import _track
+
+
+def _here() -> Path:
+    return Path.cwd() / _c.RUN_FILE
+
+
+def track(title: str, lab=None) -> int:
+    open_task = _c.task_run()
+    if open_task:
+        print(f"baseqdb: the task {open_task} is open here: finish it first (baseqdb finish).", file=sys.stderr)
+        return 1
+    _c.login()
+    run = _c.post("/runs", json={"title": title, "language": "python", "argv": ["baseqdb", "track", title],
+                                 "environment": _track.environment(), "lab": lab})
+    _here().parent.mkdir(parents=True, exist_ok=True)
+    _here().write_text(run["uid"] + "\n", encoding="utf-8")
+    print(f"baseqdb: tracking \"{title}\" as run {run['uid']}. Every command in this folder joins it; end it with "
+          "`baseqdb finish`.")
+    return 0
+
+
+def finish(errored: bool = False, finding: str | None = None, genes: list[str] | None = None) -> int:
+    uid = _c.task_run()
+    if not uid:
+        print("baseqdb: no task is open here.", file=sys.stderr)
+        return 1
+    _c.login()
+    _c.set_run(None)
+    body = {"status": "errored" if errored else "completed"}
+    if finding:
+        body["finding"] = finding
+    if genes:
+        body["genes"] = list(genes)
+    run = _c.post(f"/runs/{uid}/finish", json=body)
+    if _here().is_file() and _here().read_text(encoding="utf-8").strip() == uid:
+        _here().unlink()
+    print(f"baseqdb: run {uid} ended ({run.get('status')}).")
+    return 0
+
+
+def main(argv=None) -> int:
+    parser = argparse.ArgumentParser(prog="baseqdb", description="An AI agent's task, or yours, as one run on baSeq.")
+    sub = parser.add_subparsers(dest="command", required=True)
+    t = sub.add_parser("track", help="begin a task's run; the commands in this folder join it")
+    t.add_argument("title", help="what the task is")
+    t.add_argument("--lab", help="one of your labs to share the run with")
+    f = sub.add_parser("finish", help="end the task's run")
+    f.add_argument("--errored", action="store_true", help="the task failed")
+    f.add_argument("--finding", help="what the task found, in a short paragraph (shown marked From code)")
+    f.add_argument("--gene", action="append", default=[], metavar="GENE",
+                   help="a gene of the finding, with + or - for its direction (ZBTB16+); again for each")
+    args = parser.parse_args(argv)
+    try:
+        if args.command == "track":
+            return track(args.title, args.lab)
+        return finish(args.errored, args.finding, args.gene)
+    except _c.BaseqError as exc:
+        print(f"baseqdb: {exc}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

baseqdb-0.5.0/src/baseqdb/_client.py

@@ -0,0 +1,206 @@
+"""The connection to baSeq's API (/api/v1): your key, requests, pages of a list, downloads kept in a cache, files sent
+in parts for a dataset you add, and the run the requests belong to (an agent's task or a tracked script, _track.py)."""
+from __future__ import annotations
+
+import hashlib
+import os
+from pathlib import Path
+
+import httpx
+
+DEFAULT_URL = "https://baseq.bio"
+VERSION = "0.5.0"
+RUN_HEADER = "X-Baseq-Run"
+RUN_FILE = Path(".baseqdb") / "run"     # an agent's task, begun with `baseqdb track`, for every command in the folder
+_state: dict = {"client": None, "base": None, "run": None}
+
+
+class BaseqError(Exception):
+    """What the API refused, in its own words, with the HTTP status (None when nothing was asked yet)."""
+
+    def __init__(self, status: int | None, message: str):
+        super().__init__(message)
+        self.status = status
+
+
+def login(key: str | None = None, url: str | None = None, client: httpx.Client | None = None) -> dict:
+    """Use your API key (make one on your Account page, under API keys). Without ``key``, the BASEQ_API_KEY
+    environment variable; without ``url``, BASEQ_URL or https://baseq.bio. ``client``: an httpx.Client of your own.
+    Returns whose key it is."""
+    key = key or os.environ.get("BASEQ_API_KEY")
+    if not key:
+        raise BaseqError(None, "No API key: pass key= or set BASEQ_API_KEY. Make one on your Account page, under API keys.")
+    base = (url or os.environ.get("BASEQ_URL") or DEFAULT_URL).rstrip("/") + "/api/v1"
+    http = client or httpx.Client(timeout=httpx.Timeout(60.0, read=600.0))
+    http.headers["Authorization"] = f"Bearer {key}"
+    http.headers["User-Agent"] = f"baseqdb/{VERSION}"
+    _state.update(client=http, base=base)
+    set_run(task_run())
+    return me()
+
+
+def task_run() -> str | None:
+    """The open task this code is part of: BASEQ_RUN, or the uid in .baseqdb/run here or in a folder above."""
+    if os.environ.get("BASEQ_RUN"):
+        return os.environ["BASEQ_RUN"].strip() or None
+    here = Path.cwd()
+    for folder in (here, *here.parents):
+        found = folder / RUN_FILE
+        if found.is_file():
+            return found.read_text(encoding="utf-8").strip() or None
+    return None
+
+
+def set_run(uid: str | None) -> None:
+    """The run the requests from now on belong to (None: no run)."""
+    _state["run"] = uid
+    http = _state["client"]
+    if http is None:
+        return
+    if uid:
+        http.headers[RUN_HEADER] = uid
+    else:
+        http.headers.pop(RUN_HEADER, None)
+
+
+def current_run() -> str | None:
+    return _state["run"]
+
+
+def me() -> dict:
+    """Whose key this is, what it may do and when it ends."""
+    return get("/me")
+
+
+def _http() -> tuple[httpx.Client, str]:
+    if _state["client"] is None:
+        login()
+    return _state["client"], _state["base"]
+
+
+def _raise(response: httpx.Response) -> None:
+    if response.status_code < 400:
+        return
+    try:
+        detail = response.json().get("detail")
+    except Exception:
+        detail = response.text[:200]
+    if isinstance(detail, list):        # a request the API could not read: which parameter, and why
+        detail = "; ".join(f"{'.'.join(map(str, d.get('loc', [])))}: {d.get('msg')}" for d in detail)
+    raise BaseqError(response.status_code, f"{detail} (HTTP {response.status_code})")
+
+
+def get(path: str, **params) -> dict:
+    http, base = _http()
+    response = http.get(base + path, params={k: v for k, v in params.items() if v is not None})
+    _raise(response)
+    return response.json()
+
+
+def post(path: str, json: dict | None = None) -> dict:
+    http, base = _http()
+    response = http.post(base + path, json=json)
+    _raise(response)
+    return response.json() if response.content else {}
+
+
+def put(path: str, content: bytes, **params) -> None:
+    http, base = _http()
+    response = http.put(base + path, content=content, params={k: v for k, v in params.items() if v is not None},
+                        headers={"Content-Type": "application/octet-stream"})
+    _raise(response)
+
+
+def delete(path: str) -> None:
+    http, base = _http()
+    _raise(http.delete(base + path))
+
+
+def send(paths) -> tuple[str, dict, list]:
+    """``paths`` sent in parts to a new upload: its id, each file's SHA-256 by the id it went under ("f0", "f1", ...)
+    and those ids in order. An upload that fails on the way is thrown away."""
+    began = post("/uploads")
+    upload, size = began["upload"], int(began["part_size"])
+    sums, ids = {}, []
+    try:
+        for i, path in enumerate(paths):
+            fid, h, n = f"f{i}", hashlib.sha256(), 0
+            with open(path, "rb") as fh:
+                while True:
+                    chunk = fh.read(size)
+                    if not chunk and n:
+                        break
+                    h.update(chunk)
+                    put(f"/uploads/{upload}/files/{fid}/parts/{n}", chunk, name=Path(path).name if n == 0 else None)
+                    n += 1
+                    if len(chunk) < size:
+                        break
+            sums[fid] = h.hexdigest()
+            ids.append(fid)
+    except BaseException:
+        try:
+            delete(f"/uploads/{upload}")
+        except Exception:
+            pass
+        raise
+    return upload, sums, ids
+
+
+def pages(path: str, **params):
+    """Every item of a list the API gives in pages."""
+    offset = 0
+    while True:
+        page = get(path, offset=offset, **params)
+        yield from page["items"]
+        offset += len(page["items"])
+        if not page["items"] or offset >= page["total"]:
+            return
+
+
+def sha256_of(path: Path) -> str:
+    h = hashlib.sha256()
+    with open(path, "rb") as fh:
+        for chunk in iter(lambda: fh.read(1 << 20), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def download(path: str, target: Path, sha256: str | None = None, read: dict | None = None) -> Path:
+    """``path`` saved as ``target``, once: a file already there is kept when it has the right SHA-256 (or, when none
+    is known, at all). A download whose bytes do not match is thrown away. ``read`` names the file for a run: a copy
+    kept from before is still read by the run, and baSeq is told so."""
+    if target.exists() and (sha256 is None or sha256_of(target) == sha256):
+        if read and _state["run"]:
+            note_read(read)
+        return target
+    target.parent.mkdir(parents=True, exist_ok=True)
+    part = target.with_name(target.name + ".part")
+    http, base = _http()
+    h = hashlib.sha256()
+    with http.stream("GET", base + path) as response:
+        if response.status_code >= 400:
+            response.read()
+            _raise(response)
+        with open(part, "wb") as fh:
+            for chunk in response.iter_bytes(1 << 20):
+                fh.write(chunk)
+                h.update(chunk)
+    if sha256 and h.hexdigest() != sha256:
+        part.unlink(missing_ok=True)
+        raise BaseqError(None, f"{target.name} arrived damaged (its SHA-256 is not the dataset's); try again.")
+    os.replace(part, target)
+    return target
+
+
+def cache_dir() -> Path:
+    """Where downloads are kept: BASEQDB_CACHE, or ~/.cache/baseqdb."""
+    return Path(os.environ.get("BASEQDB_CACHE") or Path.home() / ".cache" / "baseqdb")
+
+
+def note_read(item: dict) -> None:
+    """The open run read this file from baseqdb's cache: baSeq notes it as it notes a download. Best effort."""
+    try:
+        post(f"/runs/{_state['run']}/inputs", json={"items": [item]})
+    except BaseqError as exc:
+        import logging
+        logging.getLogger("baseqdb").warning("baSeq was not told the run read %s: %s", item, exc)

baseqdb-0.5.0/src/baseqdb/_records.py

@@ -0,0 +1,367 @@
+"""baSeq's datasets, analyses, records and runs, with lamindb's names: ``get`` one, ``filter`` many, ``cache`` the
+files, ``load`` a table, ``to_dataframe`` a list. A Dataset is lamindb's Collection (its versions share a stem), a
+Record its Artifact, a Run its Run."""
+from __future__ import annotations
+
+import logging
+import os
+from datetime import datetime
+from pathlib import Path, PurePosixPath
+from urllib.parse import quote
+
+from baseqdb import _client as _c
+
+_log = logging.getLogger("baseqdb")
+
+
+def _when(value):
+    return datetime.fromisoformat(value.replace("Z", "+00:00")) if isinstance(value, str) else value
+
+
+def _pandas():
+    try:
+        import pandas
+    except ImportError:
+        raise ImportError("this needs pandas: pip install pandas") from None
+    return pandas
+
+
+def _inside(folder: Path, name: str) -> Path:
+    """``name`` (which may hold a folder, "data_summary/counts.tsv") under ``folder``, never above it."""
+    parts = [p for p in PurePosixPath(name).parts if p not in ("", ".", "..", "/")]
+    return folder.joinpath(*parts)
+
+
+class Results(list):
+    """A list of baSeq things. ``to_dataframe()`` gives their fields as a pandas DataFrame."""
+
+    def to_dataframe(self):
+        return _pandas().DataFrame([x.row() for x in self])
+
+
+class Files(Results):
+    """A dataset's files, by position or by name: ``ds.files["counts.tsv"]``."""
+
+    def __getitem__(self, key):
+        if isinstance(key, str):
+            for f in self:
+                if f.name == key:
+                    return f
+            raise KeyError(f"no file {key!r}; there are {[f.name for f in self]}")
+        return super().__getitem__(key)
+
+
+class _Thing:
+    fields: tuple = ()
+
+    def __init__(self, data: dict):
+        self._data = dict(data)
+        for f in self.fields:
+            setattr(self, f, _when(data.get(f)) if f.endswith("_at") else data.get(f))
+
+    def row(self) -> dict:
+        return {f: getattr(self, f) for f in self.fields}
+
+    def __repr__(self):
+        shown = ", ".join(f"{f}={getattr(self, f)!r}" for f in self.fields[:4])
+        return f"{type(self).__name__}({shown})"
+
+
+# ---------------------------------------------------------------------------------------------- datasets
+
+class Dataset(_Thing):
+    """A dataset: one version (``uid``) of a family (``stem``).
+
+    ``Dataset(files, name=..., kind=...)`` is one to add with ``save()``, as lamindb's ``Artifact(path, key=...)``:
+    with the name and kind of a dataset you have, it is its next version."""
+    fields = ("uid", "stem", "version", "name", "kind", "organism", "file_count", "size", "source", "created_at", "mine")
+
+    def __init__(self, data=None, *, name: str | None = None, kind: str | None = None, sample_sheet=None, lab=None):
+        if isinstance(data, dict):          # as the API describes one
+            super().__init__(data)
+            self._draft = None
+            return
+        files = [data] if isinstance(data, (str, os.PathLike)) else list(data or [])
+        if not files or not name or not kind:
+            raise TypeError("Dataset(files, name=..., kind=...): the files to add, the dataset's name and its kind "
+                            "(a kind of data your plan includes, or part of its name, such as 'bulk')")
+        super().__init__({"name": name, "kind": kind})
+        self._draft = {"files": [Path(f) for f in files], "sample_sheet": Path(sample_sheet) if sample_sheet else None,
+                       "lab": lab, "sent": None}
+        self.notes: list[str] = []
+
+    def save(self) -> "Dataset":
+        """Add it, with a key that may add data: its files sent in parts, each checked by its SHA-256 on arrival. With
+        the name and kind of a dataset you have, its next version, which starts what that dataset's "When a new version
+        is added" says; with the very files of one you have, that one. What else to know is in ``notes``. Itself."""
+        draft = getattr(self, "_draft", None)
+        if draft is None:
+            return self
+        paths = draft["files"] + ([draft["sample_sheet"]] if draft["sample_sheet"] else [])
+        for path in paths:
+            if not path.is_file():
+                raise _c.BaseqError(None, f"{path} is not a file.")
+        for attempt in (1, 2):
+            if draft["sent"] is None:
+                draft["sent"] = _c.send(paths)
+            upload, sums, ids = draft["sent"]
+            body = {"name": self.name, "kind": self.kind, "lab": draft["lab"], "sha256": sums,
+                    "sample_sheet": ids[-1] if draft["sample_sheet"] else None}
+            try:
+                answer = _c.post(f"/uploads/{upload}/finish", json=body)
+                break
+            except _c.BaseqError as exc:
+                # A refusal that can be put right keeps the upload for another save(); one gone is sent again, once
+                if exc.status in (404, 422) or attempt == 2:
+                    draft["sent"] = None
+                if exc.status != 404 or attempt == 2:
+                    raise
+        _Thing.__init__(self, answer)
+        self._draft = None
+        self.notes = list(answer.get("notes") or [])
+        for note in self.notes:
+            _log.warning(note)
+        return self
+
+    @classmethod
+    def get(cls, uid: str) -> "Dataset":
+        """By its uid (that version) or its stem (its newest version you can open)."""
+        return cls(_c.get(f"/datasets/{quote(uid)}"))
+
+    @classmethod
+    def filter(cls, kind: str | None = None, name: str | None = None, all_versions: bool = False) -> Results:
+        """The datasets you can open, newest first, each at its newest version you can open (or every version)."""
+        return Results(cls(d) for d in _c.pages("/datasets", kind=kind, name=name,
+                                                all_versions=str(bool(all_versions)).lower(), limit=1000))
+
+    def _detail(self) -> dict:
+        if "files" not in self._data:
+            self._data.update(_c.get(f"/datasets/{self.uid}"))
+        return self._data
+
+    @property
+    def files(self) -> Results:
+        return Files(File(f, self) for f in self._detail()["files"])
+
+    @property
+    def run(self) -> "Run | None":
+        """The run that made this version, as lamindb's collection.run, when you can open it: its import, whose
+        initiated_by is the run it was added in when it came from code."""
+        uid = self._detail().get("run")
+        return Run.get(uid) if uid else None
+
+    @property
+    def versions(self) -> Results:
+        """Every version of this dataset you can open, oldest first."""
+        return Results(self if v["uid"] == self.uid else Dataset.get(v["uid"]) for v in self._detail()["versions"])
+
+    @property
+    def sample_sheet(self):
+        """The sample sheet, as a pandas DataFrame (the sample first)."""
+        sheet = _c.get(f"/datasets/{self.uid}/sample-sheet")
+        return _pandas().DataFrame(sheet["rows"], columns=sheet["columns"])
+
+    @property
+    def analyses(self) -> Results:
+        """The analyses of this version you can open."""
+        return Results(Analysis(a) for a in _c.get(f"/datasets/{self.uid}/analyses")["items"])
+
+    def cache(self, folder: str | Path | None = None) -> Path:
+        """Its files, downloaded once into ``folder`` (by default baseqdb's cache, datasets/<uid>), each checked
+        against its SHA-256. The folder."""
+        root = Path(folder) if folder else _c.cache_dir() / "datasets" / self.uid
+        for f in self.files:
+            f.cache(root)
+        return root
+
+
+class File(_Thing):
+    """One of a dataset's files."""
+    fields = ("name", "size", "sha256")
+
+    def __init__(self, data: dict, dataset: Dataset):
+        super().__init__(data)
+        self.dataset = dataset
+
+    def cache(self, folder: str | Path | None = None) -> Path:
+        root = Path(folder) if folder else _c.cache_dir() / "datasets" / self.dataset.uid
+        return _c.download(f"/datasets/{self.dataset.uid}/files/{quote(self.name)}", _inside(root, self.name), self.sha256,
+                           read={"dataset": self.dataset.uid, "file": self.name})
+
+    def load(self, **read_options):
+        """A table as a pandas DataFrame, as lamindb's ``artifact.load()``: a .csv file read comma-separated, a .tsv
+        or .txt file tab-separated (``read_options`` go to pandas.read_csv). Any other file: ``cache()`` for its path."""
+        name = self.name.lower().removesuffix(".gz")
+        if not name.endswith((".csv", ".tsv", ".txt")):
+            raise _c.BaseqError(None, f"load() reads tables (.csv, .tsv, .txt); for {self.name}, cache() gives its path.")
+        sep = "," if name.endswith(".csv") else "\t"
+        return _pandas().read_csv(self.cache(), **{"sep": sep, **read_options})
+
+
+# ---------------------------------------------------------------------------------------------- analyses
+
+class Analysis(_Thing):
+    """An analysis: its steps, what they found, its significant genes and its tables."""
+    fields = ("id", "name", "kind", "project", "dataset", "organism", "created_at", "updated_at", "frozen", "mine")
+
+    @classmethod
+    def get(cls, id: int) -> "Analysis":
+        return cls(_c.get(f"/analyses/{int(id)}"))
+
+    @classmethod
+    def filter(cls, kind: str | None = None, name: str | None = None) -> Results:
+        """The analyses you can open, newest first, with the dataset each was made from when it was."""
+        return Results(cls(a) for a in _c.pages("/analyses", kind=kind, name=name, limit=1000))
+
+    def _detail(self) -> dict:
+        if "steps" not in self._data:
+            self._data.update(_c.get(f"/analyses/{self.id}"))
+        return self._data
+
+    @property
+    def steps(self) -> Results:
+        """Each step in the order it was added, with its settings and when it ran."""
+        return Results(Step(s) for s in self._detail()["steps"])
+
+    @property
+    def findings(self) -> Results:
+        """What each step found."""
+        return Results(Finding(f) for f in _c.get(f"/analyses/{self.id}/findings")["items"])
+
+    def genes(self, gene: str | None = None, step_id: int | None = None) -> Results:
+        """The significant genes of its result tables (adjusted p 0.05 or less), or one gene's."""
+        return Results(GeneHit(g) for g in _c.pages(f"/analyses/{self.id}/genes", gene=gene, step_id=step_id,
+                                                      limit=10000))
+
+    @property
+    def tables(self) -> "Tables":
+        """Its tables by name ("dge_condition_treated_vs_control.csv", "data_summary/counts.tsv"); when two steps
+        wrote one of the same name, the later step's. ``tables.all`` lists every one."""
+        return Tables(Table(t, self) for t in _c.get(f"/analyses/{self.id}/tables")["items"])
+
+
+class Tables(dict):
+    def __init__(self, tables):
+        super().__init__()
+        self.all = Results(tables)
+        for t in self.all:
+            self[t.name] = t
+
+    def __missing__(self, name):
+        raise KeyError(f"no table {name!r}; there are {sorted(self)}")
+
+    def to_dataframe(self):
+        return self.all.to_dataframe()
+
+
+class Step(_Thing):
+    fields = ("id", "method", "title", "status", "settings", "created_at", "parent", "tables")
+
+    def row(self) -> dict:
+        return {**super().row(), "run": (self._data.get("run") or {}).get("uid")}
+
+    @property
+    def run(self) -> "Run | None":
+        found = self._data.get("run")
+        return Run.get(found["uid"]) if found else None
+
+
+class Finding(_Thing):
+    fields = ("step_id", "method", "title", "finding", "genes", "terms", "conditions")
+
+
+class GeneHit(_Thing):
+    fields = ("step_id", "gene", "direction", "log2fc", "padj", "contrast", "table")
+
+
+class Table(_Thing):
+    """A table a step wrote."""
+    fields = ("step_id", "method", "name", "size")
+
+    def __init__(self, data: dict, analysis: Analysis):
+        super().__init__(data)
+        self.analysis = analysis
+
+    def cache(self, folder: str | Path | None = None) -> Path:
+        root = Path(folder) if folder else _c.cache_dir() / "analyses" / str(self.analysis.id) / str(self.step_id)
+        path = f"/analyses/{self.analysis.id}/tables/{self.step_id}/{quote(self.name)}"
+        return _c.download(path, _inside(root, self.name),
+                           read={"analysis": self.analysis.id, "step": self.step_id, "table": self.name})
+
+    def load(self, **read_options):
+        """The table as a pandas DataFrame (``read_options`` go to pandas.read_csv)."""
+        sep = "\t" if self.name.lower().endswith(".tsv") else ","
+        return _pandas().read_csv(self.cache(), **{"sep": sep, **read_options})
+
+
+# ---------------------------------------------------------------------------------------------- records and runs
+
+class Record(_Thing):
+    """A record: one version of a file the platform keeps (lamindb's Artifact).
+
+    ``Record(path).save()`` keeps a file your tracked code made with its run (bq.track(), or an agent's task), as
+    lamindb's ``Artifact(path).save()`` inside ``ln.track()``: saved again under the same name, its next version."""
+    fields = ("uid", "stem", "version", "is_latest", "name", "suffix", "role", "kind", "size", "sha256", "lamin_hash",
+              "created_at", "superseded_at", "gone_at", "analysis", "step")
+
+    def __init__(self, data=None, *, name: str | None = None):
+        if isinstance(data, dict):
+            super().__init__(data)
+            self._path = None
+            return
+        if data is None:
+            raise TypeError("Record(path): the file to keep with the run")
+        self._path = Path(data)
+        super().__init__({"name": name or self._path.name})
+
+    def save(self) -> "Record":
+        """Keep the file with the open run, checked by its SHA-256 on arrival. Itself."""
+        if self._path is None:
+            return self
+        if not _c.current_run():
+            raise _c.BaseqError(None, "A file is kept with a run: begin one with bq.track() (or `baseqdb track`) first.")
+        if not self._path.is_file():
+            raise _c.BaseqError(None, f"{self._path} is not a file.")
+        upload, sums, ids = _c.send([self._path])
+        answer = _c.post(f"/uploads/{upload}/record", json={"name": self.name, "sha256": sums[ids[0]]})
+        _Thing.__init__(self, {**answer, "is_latest": True})
+        self._path = None
+        return self
+
+    @classmethod
+    def get(cls, uid: str) -> "Record":
+        return cls(_c.get(f"/records/{quote(uid)}"))
+
+    def row(self) -> dict:
+        return {**super().row(), "run": self._data.get("run")}
+
+    @property
+    def run(self) -> "Run | None":
+        """The run that wrote it, when you can open it."""
+        return Run.get(self._data["run"]) if self._data.get("run") else None
+
+
+class Run(_Thing):
+    """A run: what ran, with which settings, in which environment, what it read and what it wrote."""
+    fields = ("uid", "kind", "status", "status_code", "transform", "settings", "created_at", "started_at",
+              "finished_at", "analysis", "step", "dataset", "initiated_by", "environment")
+
+    @classmethod
+    def get(cls, uid: str) -> "Run":
+        return cls(_c.get(f"/runs/{quote(uid)}"))
+
+    @property
+    def inputs(self) -> Results:
+        """The records it read (those you can open)."""
+        return Results(Record(r) for r in self._data.get("inputs") or [])
+
+    @property
+    def outputs(self) -> Results:
+        """The records it wrote (those you can open)."""
+        return Results(Record(r) for r in self._data.get("outputs") or [])
+
+    @property
+    def initiated_runs(self) -> Results:
+        """The runs it started, as lamindb's run.initiated_runs: an agent's task's scripts, the import of a dataset it
+        added (those you can open)."""
+        return Results(Run.get(uid) for uid in self._data.get("initiated_runs") or [])

baseqdb-0.5.0/src/baseqdb/_track.py

@@ -0,0 +1,118 @@
+"""Your code as a run on baSeq, as lamindb's ``ln.track()``: ``bq.track()`` at the top of a script makes it one run,
+with its code (kept as a version of the script), its command line, its parameters and its environment; the files it
+downloads from baSeq while it runs are what it read, what it adds was added in it, and ``bq.Record(path).save()``
+keeps a file with it. ``bq.finish()`` ends it, with what it found when you say (``finding=``, a finding from code),
+and a script that ends without it ends its run anyway: completed, or errored when an exception stopped it.
+
+Inside an agent's task (``baseqdb track "<task>"``, _cli.py) a tracked script's run is part of the task's run, and code
+that does not track itself still names the task in its requests, so whatever the task does is one run."""
+from __future__ import annotations
+
+import atexit
+import os
+import platform
+import sys
+
+from baseqdb import _client as _c
+
+SOURCE_MAX = 2 * 1024 * 1024
+_state: dict = {"run": None, "task": None, "errored": False, "hooked": False, "finished": False}
+
+
+def _script() -> tuple[str, str | None]:
+    """(the code's file name, its code): the script running; '-c' and its code for ``python -c``; '-' when there is no
+    file (standard input, an interactive session)."""
+    orig = list(getattr(sys, "orig_argv", None) or [])
+    if "-c" in orig and orig.index("-c") + 1 < len(orig):
+        return "-c", orig[orig.index("-c") + 1]
+    main = sys.modules.get("__main__")
+    path = getattr(main, "__file__", None)
+    if path and os.path.isfile(path) and os.path.getsize(path) <= SOURCE_MAX:
+        try:
+            with open(path, encoding="utf-8", errors="replace") as fh:
+                return os.path.basename(path), fh.read()
+        except OSError:
+            pass
+    return "-", None
+
+
+def environment() -> dict:
+    """Python and every installed distribution with its version, as baSeq keeps an analysis kernel's."""
+    from importlib import metadata
+    packages = {}
+    for dist in metadata.distributions():
+        name = (dist.metadata.get("Name") or "").strip()
+        if name:
+            packages.setdefault(name, dist.version)
+    return {"python": {"version": platform.python_version(), "build": sys.version, "packages": packages}}
+
+
+def imported() -> list[str]:
+    """The distributions the code imported, for the run's page."""
+    from importlib import metadata
+    try:
+        owners = metadata.packages_distributions()
+    except Exception:
+        return []
+    names = set()
+    for module in list(sys.modules):
+        top = module.split(".", 1)[0]
+        for dist in owners.get(top, []):
+            names.add(dist)
+    return sorted(names)
+
+
+def track(title: str | None = None, params: dict | None = None, lab=None) -> dict:
+    """This script as one run on baSeq (a key that may add data). ``title``: what it is, its file's name without it;
+    ``params``: the settings it ran with; ``lab``: one of your labs to share the run with. Inside an agent's task, its
+    run is part of the task's. The run, as baSeq describes it."""
+    if _state["run"] and not _state["finished"]:
+        return {"uid": _state["run"]}
+    _c._http()
+    filename, code = _script()
+    task = _c.task_run()
+    run = _c.post("/runs", json={"title": title, "language": "python", "filename": filename, "source": code,
+                                 "argv": list(sys.argv), "params": params or {}, "environment": environment(),
+                                 "lab": None if task else lab, "parent": task})
+    _state.update(run=run["uid"], task=task, errored=False, finished=False)
+    _c.set_run(run["uid"])
+    if not _state["hooked"]:
+        _state["hooked"] = True
+        previous = sys.excepthook
+
+        def hook(kind, value, tb):
+            _state["errored"] = True
+            previous(kind, value, tb)
+        sys.excepthook = hook
+        atexit.register(_at_exit)
+    print(f"baseqdb: tracking this run as {run['uid']}", file=sys.stderr)
+    return run
+
+
+def finish(status: str | None = None, finding: str | None = None, genes: list[str] | None = None) -> dict | None:
+    """End the run begun with track(): ``completed`` (or ``errored``). ``finding``: what it found, in a short paragraph,
+    with its ``genes`` ("ZBTB16+", "FKBP5+"): kept with the run (the task's, inside a task) and shown on its page and
+    in your lab's findings, marked From code. The requests after it belong to the task again, when there is one."""
+    if not _state["run"] or _state["finished"]:
+        return None
+    status = status or ("errored" if _state["errored"] else "completed")
+    body = {"status": status, "imported": imported()}
+    if finding:
+        body["finding"] = str(finding)
+    if genes:
+        body["genes"] = [str(g) for g in genes]
+    _state["finished"] = True
+    try:
+        answer = _c.post(f"/runs/{_state['run']}/finish", json=body)
+    except Exception:
+        _state["finished"] = False      # still open on baSeq: finish() again, or the end of the script ends it
+        raise
+    _c.set_run(_state["task"])
+    return answer
+
+
+def _at_exit() -> None:
+    try:
+        finish()
+    except Exception as exc:          # the interpreter is ending: say so, never raise
+        print(f"baseqdb: the run was not ended on baSeq: {exc}", file=sys.stderr)