orfdata 0.34.1__tar.gz → 0.36.0__tar.gz

{orfdata-0.34.1/orfdata.egg-info → orfdata-0.36.0}/PKG-INFO

@@ -1,17 +1,20 @@
 Metadata-Version: 2.4
 Name: orfdata
-Version: 0.34.1
+Version: 0.36.0
 Summary: Reference proteomes with evidence-backed noncanonical ORFs on PyEnsembl genomes
 Author: OpenVax
 License-Expression: Apache-2.0
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: datacache<2,>=1.25.1
+Requires-Dist: pandas<4,>=3
+Requires-Dist: pyyaml<7,>=6
 Provides-Extra: pyensembl
 Requires-Dist: pyensembl>=2.27.2; extra == "pyensembl"
 Provides-Extra: build-data
 Requires-Dist: openpyxl>=3.1; extra == "build-data"
-Requires-Dist: biopython<2,>=1.80; extra == "build-data"
+Provides-Extra: viral
 Provides-Extra: data
 Requires-Dist: datacache<2,>=1.25.1; extra == "data"
 Provides-Extra: columnar
@@ -22,9 +25,9 @@ Requires-Dist: ruff; extra == "dev"
 Requires-Dist: build; extra == "dev"
 Requires-Dist: twine; extra == "dev"
 Requires-Dist: openpyxl>=3.1; extra == "dev"
-Requires-Dist: biopython<2,>=1.80; extra == "dev"
 Requires-Dist: pyensembl>=2.27.2; extra == "dev"
 Requires-Dist: datacache<2,>=1.25.1; extra == "dev"
+Requires-Dist: gb-io<0.5,>=0.4; extra == "dev"
 Provides-Extra: docs
 Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
 Requires-Dist: mkdocs-material<10,>=9.5; extra == "docs"
@@ -35,8 +38,8 @@ Dynamic: license-file
 
 # orfdata
 
-orfdata builds the protein reference for a species: the proteins annotated in
-a [PyEnsembl](https://github.com/openvax/pyensembl) genome, plus open reading
+orfdata builds protein references from transcript models in
+a [PyEnsembl](https://github.com/openvax/pyensembl) annotation, plus open reading
 frames (ORFs) that ribosome-profiling studies found translated outside those
 annotations. These include upstream and downstream ORFs, ORFs in other reading
 frames of a coding sequence, and ORFs on RNAs annotated as noncoding. Each added
@@ -48,38 +51,75 @@ evidence.
 Use Python 3.11 or later. Install orfdata and one PyEnsembl annotation:
 
 ```sh
-python -m pip install 'orfdata[pyensembl]'
+python -m pip install 'orfdata[pyensembl,build-data]'
 pyensembl install --release 115 --species human
+# For the ncORF evidence presets and "all":
+orfdata setup --species human --catalogs all
 ```
 
+List every bundled, downloadable or buildable resource and what is installed:
+
+```sh
+orfdata download list
+orfdata download status
+```
+
+`orfdata setup --species human` also installs `gdc-viruses`, the default
+200-reference GDC viral screen for human transcriptome analysis;
+`--collections none` skips it. Viral support is part of the base install, and
+only the data is downloaded. To acquire only the viral references:
+
+```sh
+orfdata download viral --species human
+orfdata viral summary
+```
+
+Each viral collection keeps three layers apart — the viral genomes, one coding
+transcript per source CDS feature with a GTF placing it on them, and the
+proteins those transcripts encode. Because the three share identifiers, the set
+opens as a PyEnsembl genome, so varcode, isovar, topiary and vaxrank can
+annotate variants on viral contigs. `orfdata proteome --annotation DIR` writes
+the same three files for noncanonical ORFs. See
+[Downstream tools](https://openvax.github.io/orfdata/downstream/).
+
 ## Example
 
 ```python
 from pyensembl import EnsemblRelease
-from orfdata import Proteome
+from orfdata import ReferenceTranscriptome
 
 genome = EnsemblRelease(115, species="human")
-proteome = Proteome.from_genome(genome).with_extra_orfs(evidence="ms")
+transcriptome = ReferenceTranscriptome.from_genome(genome)
+reference = transcriptome.proteome()                    # Reference CDS proteins only; all isoforms.
+one_per_gene = transcriptome.proteome(isoforms="longest")
+proteome = transcriptome.proteome(nc_orf_sources="ms")          # Explicitly add MS-supported ncORFs.
 proteome.write_fasta("human-e115.fasta")
-print(proteome.summary()["kinds"], len(proteome.excluded))
+print(proteome.summary()["protein_sources"], len(proteome.excluded))
 ```
 
 ```text
-{'annotated': 245535, 'extra_orf': 2181} 5
+{'reference_cds': 245535, 'nc_orf': 2181} 5
 ```
 
-This takes every protein in the Ensembl 115 annotation and adds the 2,181
+`proteome()` defaults to reference CDS proteins, with no ncORFs or viruses.
+`isoforms="longest"` selects one protein per gene by length, with accession
+and transcript ID breaking ties. This does not assign Ensembl canonical or
+MANE status. `nc_orf_sources="ms"` takes every protein in the Ensembl 115 annotation and adds the 2,181
 bundled human ORFs that have supporting mass-spectrometry peptides and are
 encoded by an Ensembl 115 transcript. Each entry lists every transcript encoding
 it; the other 5 selected ORFs are listed in `proteome.excluded` with the
-reason. Use `evidence="ribosome"` for all 7,239 encoded ribosome-profiling ORFs,
-or a rule such as `RiboSeq(min_studies=2) & MS()` to set Ribo-seq and MS
-stringency separately (1,293 encoded ORFs).
+reason. `nc_orf_sources="ribosome"` includes Ribo-seq-supported ORFs from
+all compatible registered datasets. Use a rule such as
+`RiboSeq(min_studies=2) & MS()` to set Ribo-seq and MS stringency separately.
+A dataset name such as `"bundled"` selects every ORF in that dataset;
+a list such as `["bundled", "ncorf-mammals-human"]` combines datasets.
+`nc_orf_sources="all"` includes every compatible registered dataset, including
+optional ones. Missing datasets raise with installation instructions.
 `proteome.save(path)` writes a checksummed snapshot that `Proteome.load` reads
 offline. From the command line:
 
 ```sh
-orfdata proteome --species human --release 115 --output human-e115.fasta
+orfdata proteome --species human --release 115 --nc-orf-sources ms --output human-e115.fasta
 ```
 
 A translated ORF or a detected peptide does not by itself establish a stable,
@@ -90,7 +130,7 @@ immunopeptidomics: peptides eluted from HLA molecules.
 
 | Species | Noncanonical ORFs | Sources |
 | --- | ---: | --- |
-| Human | 7,264 bundled; 57,607 with `catalogs="all"` | GENCODE; optionally Chang et al. 2025 and Yang et al. 2024 Ribo-seq |
+| Human | 7,264 bundled; 57,607 with `nc_orf_sources="all"` | GENCODE; optionally Chang et al. 2025 and Yang et al. 2024 Ribo-seq |
 | Mouse | 48,805 | Chang et al. and Yang et al. Ribo-seq, OpenProt 2.2 MS |
 | Zebrafish | 9,759 | Yang et al. Ribo-seq |
 | C. elegans | 2,849 | Yang et al. Ribo-seq |
@@ -106,10 +146,44 @@ After `orfdata chains download` (add
 (GRCh37, NCBI36, GRCm38, NCBIM37) and T2T-CHM13 use these catalogues lifted
 to them; `orfdata genomes install t2t-chm13` installs a CHM13 annotation. Dog proteomes contain their
 PyEnsembl-annotated proteins. `load_catalog(species=...)` and
-`with_extra_orfs()` pick the catalogue for a species and raise when none is
+`with_nc_orfs("all")` select catalogues for a species and raise when none is
 available or built.
-Human evidence is never transferred to another species. Viral proteomes can be added with
-`with_viral(...)`, but only to a proteome for one of the virus's recorded hosts.
+Human evidence is never transferred to another species. Viral proteins can
+be added only to a proteome for a recorded host. To build one dictionary and
+FASTA with the reference CDS proteins, all binding catalogue ORFs and the
+bundled viruses associated with human:
+
+```python
+combined = transcriptome.proteome(nc_orf_sources="all", viral="all")
+proteins_by_id = combined.by_accession()  # Reference CDS, ncORF and viral entries.
+combined.write_fasta("human-all-proteins.fasta")
+combined.save("human-all-proteins.proteome.json.gz")  # Complete provenance.
+
+nc_only = transcriptome.proteome(reference_cds=False, nc_orf_sources="all")
+nc_only.write_fasta("human-reference-ncorfs.fasta")
+```
+
+For sample-specific altered sequences, `DerivedProtein` records the supplied
+amino acids, parent protein accession, sample, change IDs and source before
+`combined.with_derived_proteins(...)` adds them. These sequences are not
+inferred from variants or included in the reference proteome by default.
+
+For a named, checksummed sequence input with no genomic placement, use the
+general `ProteinSet`. `with_protein_set(...)` records a donor, transgene or
+other selected component; `with_viral(...)` uses the same input plus a sourced
+host association. ncORF sources with transcript coordinates and evidence stay
+as `ORFCatalog` records and produce the same `ProteinSequence` output when
+bound to a reference.
+
+```sh
+orfdata proteome --species human --release 115 --nc-orf-sources all --viral all --output human-all-proteins.fasta
+```
+
+`viral="all"` selects one protein set per bundled virus and keeps its exact
+reference ID in FASTA headers. To select one virus, use `viral="HPV16"` or
+`with_viral("HPV16")`.
+Viral proteins are included only when requested; a recorded host association
+does not say which virus is present in a sample.
 orfdata also provides [RNA, ribosome and protein measurements](https://openvax.github.io/orfdata/measurements/)
 for human, mouse, dog and yeast.
 
@@ -120,10 +194,14 @@ packaged bases and coding paths. The functional loaders remain available.
 When annotating an existing alignment, `viral_orfs(reference_dna,
 contig="HPV16")` checks its actual reference sequence before applying the same
 models.
-The first [coding reference](https://openvax.github.io/orfdata/viral/) contains
-the nine RefSeq NC_001526.4 HPV16 CDSs, with native and GDC placements;
-the host association is distinct from the virus's own taxon. Broader
-imports remain in [the viral roadmap](https://github.com/openvax/orfdata/issues/91).
+The curated [coding reference](https://openvax.github.io/orfdata/viral/)
+contains the nine RefSeq NC_001526.4 HPV16 CDSs, with native and GDC
+placements; the host association is distinct from the virus's own taxon. The
+separate `gdc-viruses` source bundle contains all 200 GDC source genomes and
+their annotated CDS/protein FASTAs for human analysis. It does not assert a
+human source host for every virus; source-host metadata remains separate.
+Promoting further source records into the curated
+`ViralReference` API remains in the [viral roadmap](https://github.com/openvax/orfdata/issues/91).
 
 ## Measurement datasets
 
@@ -155,6 +233,8 @@ The [documentation](https://openvax.github.io/orfdata/) covers:
 
 | Task | Guide |
 | --- | --- |
+| Discover, inspect and download data | [Discover and download data](https://openvax.github.io/orfdata/downloads/) |
+| Use ORFs and viral references with varcode, isovar, topiary and vaxrank | [Downstream tools](https://openvax.github.io/orfdata/downstream/) |
 | Build a proteome: evidence levels, exclusions, contigs, viral proteins | [Build a reference proteome](https://openvax.github.io/orfdata/proteome/) |
 | ORF classes, evidence definitions, Varcode and Isovar integration | [ORFs and evidence](https://openvax.github.io/orfdata/reference/) |
 | Species and reference identity, mitochondrial genetic codes | [Species, references and genetic codes](https://openvax.github.io/orfdata/species/) |

{orfdata-0.34.1 → orfdata-0.36.0}/README.md

@@ -3,8 +3,8 @@
 
 # orfdata
 
-orfdata builds the protein reference for a species: the proteins annotated in
-a [PyEnsembl](https://github.com/openvax/pyensembl) genome, plus open reading
+orfdata builds protein references from transcript models in
+a [PyEnsembl](https://github.com/openvax/pyensembl) annotation, plus open reading
 frames (ORFs) that ribosome-profiling studies found translated outside those
 annotations. These include upstream and downstream ORFs, ORFs in other reading
 frames of a coding sequence, and ORFs on RNAs annotated as noncoding. Each added
@@ -16,38 +16,75 @@ evidence.
 Use Python 3.11 or later. Install orfdata and one PyEnsembl annotation:
 
 ```sh
-python -m pip install 'orfdata[pyensembl]'
+python -m pip install 'orfdata[pyensembl,build-data]'
 pyensembl install --release 115 --species human
+# For the ncORF evidence presets and "all":
+orfdata setup --species human --catalogs all
 ```
 
+List every bundled, downloadable or buildable resource and what is installed:
+
+```sh
+orfdata download list
+orfdata download status
+```
+
+`orfdata setup --species human` also installs `gdc-viruses`, the default
+200-reference GDC viral screen for human transcriptome analysis;
+`--collections none` skips it. Viral support is part of the base install, and
+only the data is downloaded. To acquire only the viral references:
+
+```sh
+orfdata download viral --species human
+orfdata viral summary
+```
+
+Each viral collection keeps three layers apart — the viral genomes, one coding
+transcript per source CDS feature with a GTF placing it on them, and the
+proteins those transcripts encode. Because the three share identifiers, the set
+opens as a PyEnsembl genome, so varcode, isovar, topiary and vaxrank can
+annotate variants on viral contigs. `orfdata proteome --annotation DIR` writes
+the same three files for noncanonical ORFs. See
+[Downstream tools](https://openvax.github.io/orfdata/downstream/).
+
 ## Example
 
 ```python
 from pyensembl import EnsemblRelease
-from orfdata import Proteome
+from orfdata import ReferenceTranscriptome
 
 genome = EnsemblRelease(115, species="human")
-proteome = Proteome.from_genome(genome).with_extra_orfs(evidence="ms")
+transcriptome = ReferenceTranscriptome.from_genome(genome)
+reference = transcriptome.proteome()                    # Reference CDS proteins only; all isoforms.
+one_per_gene = transcriptome.proteome(isoforms="longest")
+proteome = transcriptome.proteome(nc_orf_sources="ms")          # Explicitly add MS-supported ncORFs.
 proteome.write_fasta("human-e115.fasta")
-print(proteome.summary()["kinds"], len(proteome.excluded))
+print(proteome.summary()["protein_sources"], len(proteome.excluded))
 ```
 
 ```text
-{'annotated': 245535, 'extra_orf': 2181} 5
+{'reference_cds': 245535, 'nc_orf': 2181} 5
 ```
 
-This takes every protein in the Ensembl 115 annotation and adds the 2,181
+`proteome()` defaults to reference CDS proteins, with no ncORFs or viruses.
+`isoforms="longest"` selects one protein per gene by length, with accession
+and transcript ID breaking ties. This does not assign Ensembl canonical or
+MANE status. `nc_orf_sources="ms"` takes every protein in the Ensembl 115 annotation and adds the 2,181
 bundled human ORFs that have supporting mass-spectrometry peptides and are
 encoded by an Ensembl 115 transcript. Each entry lists every transcript encoding
 it; the other 5 selected ORFs are listed in `proteome.excluded` with the
-reason. Use `evidence="ribosome"` for all 7,239 encoded ribosome-profiling ORFs,
-or a rule such as `RiboSeq(min_studies=2) & MS()` to set Ribo-seq and MS
-stringency separately (1,293 encoded ORFs).
+reason. `nc_orf_sources="ribosome"` includes Ribo-seq-supported ORFs from
+all compatible registered datasets. Use a rule such as
+`RiboSeq(min_studies=2) & MS()` to set Ribo-seq and MS stringency separately.
+A dataset name such as `"bundled"` selects every ORF in that dataset;
+a list such as `["bundled", "ncorf-mammals-human"]` combines datasets.
+`nc_orf_sources="all"` includes every compatible registered dataset, including
+optional ones. Missing datasets raise with installation instructions.
 `proteome.save(path)` writes a checksummed snapshot that `Proteome.load` reads
 offline. From the command line:
 
 ```sh
-orfdata proteome --species human --release 115 --output human-e115.fasta
+orfdata proteome --species human --release 115 --nc-orf-sources ms --output human-e115.fasta
 ```
 
 A translated ORF or a detected peptide does not by itself establish a stable,
@@ -58,7 +95,7 @@ immunopeptidomics: peptides eluted from HLA molecules.
 
 | Species | Noncanonical ORFs | Sources |
 | --- | ---: | --- |
-| Human | 7,264 bundled; 57,607 with `catalogs="all"` | GENCODE; optionally Chang et al. 2025 and Yang et al. 2024 Ribo-seq |
+| Human | 7,264 bundled; 57,607 with `nc_orf_sources="all"` | GENCODE; optionally Chang et al. 2025 and Yang et al. 2024 Ribo-seq |
 | Mouse | 48,805 | Chang et al. and Yang et al. Ribo-seq, OpenProt 2.2 MS |
 | Zebrafish | 9,759 | Yang et al. Ribo-seq |
 | C. elegans | 2,849 | Yang et al. Ribo-seq |
@@ -74,10 +111,44 @@ After `orfdata chains download` (add
 (GRCh37, NCBI36, GRCm38, NCBIM37) and T2T-CHM13 use these catalogues lifted
 to them; `orfdata genomes install t2t-chm13` installs a CHM13 annotation. Dog proteomes contain their
 PyEnsembl-annotated proteins. `load_catalog(species=...)` and
-`with_extra_orfs()` pick the catalogue for a species and raise when none is
+`with_nc_orfs("all")` select catalogues for a species and raise when none is
 available or built.
-Human evidence is never transferred to another species. Viral proteomes can be added with
-`with_viral(...)`, but only to a proteome for one of the virus's recorded hosts.
+Human evidence is never transferred to another species. Viral proteins can
+be added only to a proteome for a recorded host. To build one dictionary and
+FASTA with the reference CDS proteins, all binding catalogue ORFs and the
+bundled viruses associated with human:
+
+```python
+combined = transcriptome.proteome(nc_orf_sources="all", viral="all")
+proteins_by_id = combined.by_accession()  # Reference CDS, ncORF and viral entries.
+combined.write_fasta("human-all-proteins.fasta")
+combined.save("human-all-proteins.proteome.json.gz")  # Complete provenance.
+
+nc_only = transcriptome.proteome(reference_cds=False, nc_orf_sources="all")
+nc_only.write_fasta("human-reference-ncorfs.fasta")
+```
+
+For sample-specific altered sequences, `DerivedProtein` records the supplied
+amino acids, parent protein accession, sample, change IDs and source before
+`combined.with_derived_proteins(...)` adds them. These sequences are not
+inferred from variants or included in the reference proteome by default.
+
+For a named, checksummed sequence input with no genomic placement, use the
+general `ProteinSet`. `with_protein_set(...)` records a donor, transgene or
+other selected component; `with_viral(...)` uses the same input plus a sourced
+host association. ncORF sources with transcript coordinates and evidence stay
+as `ORFCatalog` records and produce the same `ProteinSequence` output when
+bound to a reference.
+
+```sh
+orfdata proteome --species human --release 115 --nc-orf-sources all --viral all --output human-all-proteins.fasta
+```
+
+`viral="all"` selects one protein set per bundled virus and keeps its exact
+reference ID in FASTA headers. To select one virus, use `viral="HPV16"` or
+`with_viral("HPV16")`.
+Viral proteins are included only when requested; a recorded host association
+does not say which virus is present in a sample.
 orfdata also provides [RNA, ribosome and protein measurements](https://openvax.github.io/orfdata/measurements/)
 for human, mouse, dog and yeast.
 
@@ -88,10 +159,14 @@ packaged bases and coding paths. The functional loaders remain available.
 When annotating an existing alignment, `viral_orfs(reference_dna,
 contig="HPV16")` checks its actual reference sequence before applying the same
 models.
-The first [coding reference](https://openvax.github.io/orfdata/viral/) contains
-the nine RefSeq NC_001526.4 HPV16 CDSs, with native and GDC placements;
-the host association is distinct from the virus's own taxon. Broader
-imports remain in [the viral roadmap](https://github.com/openvax/orfdata/issues/91).
+The curated [coding reference](https://openvax.github.io/orfdata/viral/)
+contains the nine RefSeq NC_001526.4 HPV16 CDSs, with native and GDC
+placements; the host association is distinct from the virus's own taxon. The
+separate `gdc-viruses` source bundle contains all 200 GDC source genomes and
+their annotated CDS/protein FASTAs for human analysis. It does not assert a
+human source host for every virus; source-host metadata remains separate.
+Promoting further source records into the curated
+`ViralReference` API remains in the [viral roadmap](https://github.com/openvax/orfdata/issues/91).
 
 ## Measurement datasets
 
@@ -123,6 +198,8 @@ The [documentation](https://openvax.github.io/orfdata/) covers:
 
 | Task | Guide |
 | --- | --- |
+| Discover, inspect and download data | [Discover and download data](https://openvax.github.io/orfdata/downloads/) |
+| Use ORFs and viral references with varcode, isovar, topiary and vaxrank | [Downstream tools](https://openvax.github.io/orfdata/downstream/) |
 | Build a proteome: evidence levels, exclusions, contigs, viral proteins | [Build a reference proteome](https://openvax.github.io/orfdata/proteome/) |
 | ORF classes, evidence definitions, Varcode and Isovar integration | [ORFs and evidence](https://openvax.github.io/orfdata/reference/) |
 | Species and reference identity, mitochondrial genetic codes | [Species, references and genetic codes](https://openvax.github.io/orfdata/species/) |

{orfdata-0.34.1 → orfdata-0.36.0}/docs/catalogs.md

@@ -5,8 +5,9 @@ downloads. `CatalogRegistry` manages additional versioned ORF catalogues, with
 the same explicit discovery, download, build and offline-open workflow as
 `DatasetRegistry`. Once a species' catalogue in the packaged registry is
 downloaded or built, `load_catalog(species=...)` opens it, so `extra_orfs(genome)` and
-`Proteome.with_extra_orfs()` use it without `catalog=`; pass
-`load_catalog(species=..., registry=...)` as `catalog=` for your own registry.
+`Proteome.with_nc_orfs("all")` use registered sources. Pass
+`load_catalog(species=..., registry=...)` as `nc_orf_sources=` to a proteome
+constructor for your own registry.
 Every catalogue for the species is used, merged: those on the requested
 assembly as they are, and those on another assembly the pinned chains reach,
 lifted ([details](#every-assembly-of-a-species)). Each catalogue contributes its
@@ -181,14 +182,14 @@ build the catalogue from OpenProt's own downloads (3.4 MB, about 5 seconds),
 each checked against its pinned size and SHA-256, with `orfdata setup --species
 mouse` along with the Ribo-seq catalogue above, or alone with `orfdata catalogs
 build openprot-mouse`. `load_catalog(species="mouse")`, `extra_orfs(genome)`
-and `Proteome.build(genome)` use all three mouse catalogues, for any GRCm39
-Ensembl release, so all must be installed. With only MS-supported ORFs selected, the default:
+and `Proteome.build(genome, nc_orf_sources="ms")` use all three mouse catalogues, for any GRCm39
+Ensembl release, so all must be installed. To add only MS-supported ORFs:
 
 ```python
 from pyensembl import EnsemblRelease
 from orfdata import Proteome
 
-proteome = Proteome.build(EnsemblRelease(115, species="mouse"))
+proteome = Proteome.build(EnsemblRelease(115, species="mouse"), nc_orf_sources="ms")
 print(proteome.summary()["kinds"], len(proteome.excluded))
 ```
 
@@ -245,7 +246,7 @@ characterization of alternative proteins.* Nucleic Acids Res. 2024;52(D1):D522
 
 ## Human Ribo-seq ORFs (optional)
 
-Human's default is the bundled GENCODE catalogue: 7,264 ORFs, offline, with
+`load_catalog(species="human")` defaults to the bundled GENCODE catalogue: 7,264 ORFs, offline, with
 no setup. Two larger Ribo-seq catalogues are optional, used only when asked
 for, because they would change every human selection and make loading human
 ORFs about 60 times slower:
@@ -259,11 +260,13 @@ from orfdata import Proteome, extra_orfs, load_catalog
 
 catalog = load_catalog(species="human", catalogs="all")   # Bundled, Chang et al. and Yang et al., merged
 orfs = extra_orfs(genome, "ribosome", catalogs="all")
-proteome = Proteome.build(genome, "all", catalogs="all")
+proteome = Proteome.build(genome, nc_orf_sources="all")
 load_catalog(species="human", catalogs=["ncorf-mammals-human"])   # Just one
 ```
 
-`orfdata proteome --catalogs all` does the same. They merge after the bundled
+`orfdata proteome --nc-orf-sources all` selects every dataset. Evidence presets
+such as `nc_orf_sources="ms"` also query every compatible dataset; use
+`nc_orf_sources="bundled"` for only the bundled set. They merge after the bundled
 catalogue, Chang et al.'s before Yang et al.'s as for mouse, so an ORF in
 several keeps the first one's ID. Together they give 57,607 ORFs, of which 57,485 place on Ensembl 115; that
 takes about a minute and 3.2 GB of memory, against about a second for the
@@ -342,7 +345,7 @@ No public resource gives dog ORF-level translation evidence, as of October
 - **Mass spectrometry.** Every dog dataset searched canonical proteins only,
   so none reports a noncanonical peptide as deposited: the Canine
   PeptideAtlas (PXD052932, 149,546 peptides from 13 tissues, plasma and
-  urine), the dog tumour DLA immunopeptidome (PXD074485, 12,949 distinct
+  urine), the dog tumour DLA immunopeptidome (PXD074485, 12,948 distinct
   database-search ligands), the Pfizer TMT atlas (PXD070290) and the dog brain
   proteome (PXD033420, PXD034449). The first two are available as
   [dog peptide datasets](dog-peptides.md), as evidence for known proteins.
@@ -356,7 +359,7 @@ abundance and peptides are available as [measurements](measurements.md).
 
 Catalogues on another assembly of the species are lifted automatically. Download
 the pinned chain files once, and `load_catalog`, `extra_orfs(genome)` and
-`Proteome.build(genome)` work on every human and mouse assembly PyEnsembl
+`Proteome.build(genome, nc_orf_sources="ms")` work on every human and mouse assembly PyEnsembl
 knows:
 
 ```sh
@@ -367,7 +370,7 @@ orfdata chains download --accept-terms   # Every pinned chain; without --accept-
 from pyensembl import EnsemblRelease
 from orfdata import Proteome
 
-Proteome.build(EnsemblRelease(75))       # GRCh37: the GRCh38 catalogue, lifted
+Proteome.build(EnsemblRelease(75), nc_orf_sources="ms")  # GRCh37: the GRCh38 catalogue, lifted
 ```
 
 With evidence `None` (every catalogue ORF), each assembly gets:
@@ -399,7 +402,7 @@ silently dropped.
   each with the command that installs it; nothing is downloaded while
   loading. `cache_root` (or `--cache-root`) is where both are looked for.
 - **Your own catalogues.** `extra_orfs(genome, catalog=...)` and
-  `Proteome.build(genome, catalog=...)` lift a catalogue, or each of a list,
+  `Proteome.build(genome, nc_orf_sources=catalog)` lift a catalogue, or each of a list,
   that is on another assembly of the species through the same chains.
 - **Other names.** Assemblies go by Ensembl's names. `hg38`, `GRCh38.p14`,
   `mm39` and the like get an error that names the assembly to use, and a
@@ -427,14 +430,14 @@ GRCh38-to-CHM13 chain is public domain (CC0):
 ```sh
 orfdata genomes install t2t-chm13          # 143 MB of Ensembl files, plus about 1.1 GB of PyEnsembl index
 orfdata chains download GRCh38 T2T-CHM13v2.0
-orfdata proteome --genome t2t-chm13 --output chm13.fasta
+orfdata proteome --genome t2t-chm13 --nc-orf-sources ms --output chm13.fasta
 ```
 
 ```python
 from orfdata import Proteome, load_genome
 
 genome = load_genome("t2t-chm13")          # A PyEnsembl Genome named T2T-CHM13v2.0, offline
-proteome = Proteome.build(genome)          # 104,957 annotated proteins + 2,039 MS-supported ORFs
+proteome = Proteome.build(genome, nc_orf_sources="ms")  # 104,957 reference proteins + 2,039 MS-supported ORFs
 ```
 
 The bundled GRCh38 catalogue lifts automatically: 7,208 of 7,264 ORFs lift, and
@@ -471,7 +474,7 @@ orfdata lift --species human --to GRCh37 --output orfs-grch37.json.gz
 `orfdata lift` writes the lifted catalogue and prints a summary to standard
 error. `--catalog FILE` lifts a local catalogue instead, and `--chain FILE` uses
 your own chain file instead of a downloaded pin. Pass the result back with
-`--catalog`, for example to `orfdata proteome`; JSON and TSV exports include
+`--nc-orf-sources` for `orfdata proteome`; JSON and TSV exports include
 each record's original and the chain. Downloading chains needs the
 `orfdata[data]` extra, which adds DataCache. In Python:
 

{orfdata-0.34.1 → orfdata-0.36.0}/docs/dog-peptides.md

@@ -35,18 +35,49 @@ searched against human SwissProt, and their peptides are human proteins, so
 they sit in the dataset's separate human reference context.
 
 Database search finds 11,363 distinct dog peptides in the UniProt searches and
-12,949 across all four databases. For the cell lines, the counts of unique
-8–14-mers from database search (591 for DLA-88\*003:02, 1,465 for \*501:01) match
-the article's. The article's tumour counts come from its curated supplementary
+12,948 across all four databases, leaving out iRT standards (below). For the
+cell lines, the counts of unique 8–14-mers from database search (591 for
+DLA-88\*003:02, 1,465 for \*501:01) match the article's. The article's tumour counts come from its curated supplementary
 lists, which aren't imported (they are CC BY-NC-ND), and match no single table.
 
 Every row keeps its PEAKS identification type in provenance as `found_by`:
 `DB Search`, `DeepNovo` (de novo sequencing) or `Homolog` (SPIDER homology
 search). Only database-search peptides are guaranteed to occur in a listed
-protein, so filter on it for a conservative ligand list. Rows of the Biognosys
-iRT retention-time standards and their fragments (2,242) are marked
-`irt_standard` in every search. That includes the merged database, which lacks
-the kit, so PEAKS named dog proteins for them there.
+protein, so filter on it for a conservative ligand list.
+
+### iRT standards
+
+Every run was spiked with Biognosys iRT retention-time peptides, and their
+spectra appear in every search, often in disguise. 10,629 rows are marked
+`irt_standard`, and `irt_rule` names the first rule that marked them:
+
+| `irt_rule` | Rows | Marks a row that |
+|---|---|---|
+| `irt_sequence` | 1,179 | is a kit peptide or a fragment of one, with I and L equal |
+| `kit_only` | 1,097 | PEAKS attributes only to the kit (homology reads of kit peptides) |
+| `irt_precursor` | 1,276 | sits at the precursor of a row marked by the first two rules in the same immunoprecipitation: same charge, m/z within 20 ppm, retention time within a minute |
+| `de_novo_irt` | 7,077 | is a de novo sequence that PEAKS matches to the kit, or that shares 6 or more consecutive residues with a kit peptide |
+
+The precursor rule applies to de novo and homology rows. It applies to
+database-search rows only in the merged database, which lacks the kit, and only
+when no search with the kit found the same peptide. Elsewhere PEAKS could
+choose the kit peptide and didn't.
+
+- **De novo rows:** half of them (8,448 of 16,830) are iRT reads, often a
+  kit peptide with residues added or changed (`KATLFLQFGAQGSPFLK`). Don't take
+  an unmarked de novo sequence for a new ligand without checking its spectrum.
+- **Homology rows:** 8 are marked by precursor, such as `TGFLLDPVLNR`, which
+  has exactly the mass of `GTFIIDPGGVIR` and co-elutes with it.
+- **Database-search rows:** apart from 1,075 kit peptides, only one is marked:
+  `IFLAFQQKAYVAP`, matched to dog proteins in the merged database at the
+  precursor of `LFLQFGAAGSPFLK`.
+- **Chance matches:** few. Shuffled kit sequences share 6 residues with about
+  3 de novo rows, and shifting the kit's retention times by 10–20 minutes puts
+  about 14 de novo rows at a kit precursor.
+
+A mapping is `unmapped` when any of its rows is marked. Release 2026-10-16.1
+(importer `kaabinejadian-dla-v1`) marked only exact kit sequences and kit-only
+attributions (2,242 rows), so it missed the disguised reads.
 
 ```python
 from orfdata import DatasetRegistry, ScopedTarget
@@ -64,6 +95,10 @@ for m in ligands.measurements(target=peptide, quantity="spectrum_count"):
 lola = {m.target.identifier for m in ligands.measurements(
             reference=dog, assay_id="Lola-BB76:UniProt9615", quantity="spectrum_count")
         if m.provenance["found_by"] == "DB Search" and not m.provenance["irt_standard"]}
+
+# Why a row is marked as an iRT standard.
+for m in ligands.measurements(target=peptide, quantity="spectrum_count"):
+    print(m.assay_id, m.provenance["irt_standard"], m.provenance["irt_rule"])
 ```
 
 Targets are unmodified sequences; each row's modified form (for example
@@ -74,10 +109,25 @@ search and identification type, the proteins PEAKS reported for a peptide:
 proteins stay in the mapping's evidence. The search FASTAs weren't deposited,
 so no accession can be checked against a sequence.
 
-Each row also records `raw_file_deposited`. For 1,695 rows the best spectrum
-comes from a run missing from the deposit: Lily's second injection (22-1107,
-299 rows) and the first DLA-88\*012:01 runs (20240929, 1,396 rows). The raw
-files of 163828A's H58A run have no deposited results and aren't included.
+### Runs missing from the deposit
+
+PEAKS reports, for each peptide, the raw file and scan of its best spectrum,
+and sums its spectra and area over all of the sample's runs. The authors' PEAKS
+projects for two immunoprecipitations included runs that the deposit lacks, so
+each row records `raw_file_deposited`. For 1,695 rows the best spectrum comes
+from a missing run:
+
+- **Lily (H58A):** a second injection of the remaining eluate, 22-1107 "1uL
+  Residue" (fractions 12–30), 299 rows. The first injection, 22-1104, is
+  deposited.
+- **DLA-88\*012:01:** the 20240929 acquisition (fractions 14–25), 1,396 of its
+  1,635 rows. The line's archive holds only its 20241013 runs (fractions 26–28
+  in triplicate, and blanks), though the SDRF names the 20240929 run as the
+  assay.
+
+Those identifications come from the authors' tables, but their spectra can't
+be checked against the deposit. The raw files of 163828A's H58A run have no
+deposited results and aren't included.
 
 ## Canine PeptideAtlas