@pikaa-ai/pikaa 0.3.1 → 0.3.4

package/skills/pysam/references/migration_to_0_24.md

@@ -1,177 +0,0 @@
-# Migration to Pysam 0.24
-
-Use this checklist when moving an existing environment or pipeline to
-pysam 0.24.0.
-
-## Runtime Baseline
-
-- Pysam 0.24.0 was released 27 April 2026.
-- It was tested with Python 3.8–3.14.
-- It wraps HTSlib/samtools/bcftools 1.23.1.
-- It requires Cython 3 when building/extending from source.
-- The release notes expect 0.24 to be the final line supporting Python 3.8.
-
-For a reproducible environment:
-
-```bash
-uv pip install "pysam==0.24.0"
-```
-
-If maintaining Python/Cython extensions that cimport pysam declarations,
-rebuild them against 0.24. Do not reuse binaries built against a different
-pysam ABI without verification.
-
-## CRAM Behavior
-
-### Output version
-
-New CRAM output defaults to CRAM 3.1. If downstream software only supports
-3.0:
-
-```python
-pysam.AlignmentFile(
-    "output.cram",
-    "wc",
-    header=header,
-    reference_filename="reference.fa",
-    format_options=["version=3.0"],
-)
-```
-
-Upgrade downstream tooling instead of forcing 3.0 when possible.
-
-### Reference lookup
-
-Implicit EBI reference fetching is no longer enabled by default. Existing
-pipelines that relied on hidden network retrieval can now fail.
-
-Preferred fix:
-
-```python
-pysam.AlignmentFile(
-    "input.cram",
-    "rc",
-    reference_filename="reference.fa",
-)
-```
-
-Only configure `REF_PATH` and `REF_CACHE` when managed checksum-based lookup is
-intentional. Do not restore remote fetching without documenting network,
-cache, and provenance behavior.
-
-## `format_options`
-
-Pysam 0.24 fixes Python 3 handling so `AlignmentFile(format_options=...)` and
-`HTSFile.add_hts_options()` accept documented `str` values rather than
-requiring bytes:
-
-```python
-format_options=["version=3.0"]
-```
-
-Remove workarounds that encode these strings to bytes.
-
-## CIGAR Constants
-
-Pysam 0.23.2 restored top-level aliases such as `pysam.CMATCH` after a Cython
-behavior change, but the release notes warn that a future release will remove
-them.
-
-Migrate new and maintained code:
-
-```python
-# Compatibility alias
-pysam.CMATCH
-
-# Preferred
-pysam.CIGAR_OPS.CMATCH
-```
-
-Audit all CIGAR operators, not only `CMATCH`.
-
-## Modified Bases
-
-Pysam 0.24:
-
-- removes the prior five-modification-type limit in
-  `AlignedSegment.modified_bases`
-- fixes a crash for degenerate MM fields that describe no modified bases
-
-Retest MM/ML parsing with:
-
-- multiple modification codes
-- empty/degenerate MM tags
-- missing ML tags
-- forward and reverse alignments
-- `modified_bases_forward`
-
-## Quality Conversion
-
-`pysam.array_to_qualitystring()` was optimized substantially in 0.24. Remove
-custom micro-optimizations only after confirming identical handling of missing
-or invalid values.
-
-## Coverage and Python 3.13+
-
-The 0.24 release fixes coverage functionality for Python 3.13 and later.
-Re-run `count_coverage()` regression cases when upgrading both Python and
-pysam:
-
-- zero-depth positions
-- base-quality threshold boundaries
-- duplicate/secondary/supplementary filtering
-- ambiguous query bases
-- contig boundaries
-
-## Variant Documentation and Errors
-
-Pysam 0.24 adds documentation for many `VariantHeader` and `VariantRecord`
-internal field wrappers. Prefer documented public mappings and methods over
-introspection into Cython implementation details.
-
-Recent 0.23.x releases also improved `AlignmentFile` and `VariantFile` I/O
-exception handling. Catch specific `OSError`/`ValueError` conditions and do
-not depend on old message text.
-
-## Avoid Pysam 0.23.1 for Cython Extensions
-
-Pysam 0.23.1 was yanked because it broke binary compatibility for Cython
-projects by changing `AlignedSegment` size. Pysam 0.23.2 restored
-compatibility. Pure Python users were not affected in the same way.
-
-When migrating from 0.23:
-
-- skip 0.23.1
-- rebuild extension modules
-- test Cython cimports against the 0.24 declarations
-
-## Dispatcher Output
-
-Keep wrapped command output out of memory for bulk or binary operations:
-
-```python
-import pysam.samtools
-
-pysam.samtools.sort(
-    "-o",
-    "sorted.bam",
-    "input.bam",
-    catch_stdout=False,
-)
-```
-
-This is not new to 0.24, but it is important when updating older examples that
-omit `catch_stdout=False`.
-
-## Regression Checklist
-
-1. Print and record `pysam.__version__` and `pysam.__samtools_version__`.
-2. Open representative BAM, CRAM, VCF.gz, BCF, FASTA, FASTQ, and tabix files.
-3. Compare numeric queries with equivalent region strings.
-4. Decode CRAM with no network and an explicit reference.
-5. Write/read CRAM 3.1; test 3.0 only if required.
-6. Rebuild all Cython extensions.
-7. Exercise pileup and `count_coverage()` thresholds.
-8. Test modified-base MM/ML edge cases.
-9. Run sort/index/normalization dispatchers with file output.
-10. Recreate indexes and compare known regions and aggregate counts.

package/skills/pysam/references/sequence_files.md

@@ -1,316 +0,0 @@
-# FASTA, FASTQ, and Tabix-Indexed Files
-
-This reference targets pysam 0.24.0.
-
-## Indexed FASTA
-
-`FastaFile` provides random access through a faidx index.
-
-```python
-import pysam
-
-pysam.faidx("reference.fa")
-
-with pysam.FastaFile("reference.fa") as fasta:
-    print(fasta.references)
-    print(fasta.lengths)
-    print(fasta.get_reference_length("chr1"))
-```
-
-An uncompressed FASTA needs `<name>.fai`. A BGZF-compressed FASTA also needs a
-`.gzi` compressed-offset index. Ordinary gzip is not suitable for indexed
-random access.
-
-The constructor can use nonstandard index paths:
-
-```python
-fasta = pysam.FastaFile(
-    "reference.fa.gz",
-    filepath_index="indexes/reference.fa.gz.fai",
-    filepath_index_compressed="indexes/reference.fa.gz.gzi",
-)
-```
-
-### Numeric and string coordinates
-
-```python
-with pysam.FastaFile("reference.fa") as fasta:
-    # Numeric: 0-based, half-open
-    sequence = fasta.fetch("chr1", 999, 1_099)
-
-    # Region string: 1-based, inclusive
-    same_sequence = fasta.fetch(region="chr1:1000-1099")
-```
-
-If start or end is omitted, pysam uses the sequence boundary. Invalid regions
-raise `ValueError` or `IndexError`; do not silently clip unless that is the
-documented workflow.
-
-### Fetch variant context
-
-```python
-def variant_context(
-    fasta: pysam.FastaFile,
-    contig: str,
-    pos_1based: int,
-    ref: str,
-    flank: int = 20,
-) -> tuple[str, bool]:
-    start = pos_1based - 1
-    context_start = max(0, start - flank)
-    context_stop = min(
-        fasta.get_reference_length(contig),
-        start + len(ref) + flank,
-    )
-    context = fasta.fetch(contig, context_start, context_stop)
-    observed_ref = fasta.fetch(contig, start, start + len(ref))
-    return context, observed_ref.upper() == ref.upper()
-```
-
-Validate the entire REF allele, not only its first base. A mismatch often
-means the VCF and FASTA use different assemblies, contig aliases, or
-normalization.
-
-### Strand-aware extraction
-
-Coordinates do not encode strand. Reverse-complement after fetching:
-
-```python
-IUPAC_COMPLEMENT = str.maketrans(
-    "ACGTRYMKBDHVNacgtrymkbdhvn",
-    "TGCAYRKMVHDBNtgcayrkmvhdbn",
-)
-
-
-def reverse_complement(sequence: str) -> str:
-    return sequence.translate(IUPAC_COMPLEMENT)[::-1]
-
-
-sequence = fasta.fetch("chr1", start, stop)
-if strand == "-":
-    sequence = reverse_complement(sequence)
-```
-
-Confirm annotation coordinates before conversion: BED is normally 0-based
-half-open; GFF/GTF text is normally 1-based inclusive.
-
-## Sequential FASTA and FASTQ
-
-`FastxFile` streams FASTA, FASTQ, or mixed FASTX records. It does not implement
-random access.
-
-```python
-with pysam.FastxFile("reads.fastq.gz") as reads:
-    for record in reads:
-        print(record.name)
-        print(record.sequence)
-        print(record.comment)
-        print(record.quality)
-```
-
-The current class is `FastxFile`; `FastqFile` is an old compatibility name.
-
-### Record persistence
-
-`persist=True` is the default and copies each record so it remains valid.
-For high-throughput scans, `persist=False` avoids the copy:
-
-```python
-with pysam.FastxFile("reads.fastq.gz", persist=False) as reads:
-    for record in reads:
-        process(record.name, record.sequence)
-        # Do not save `record` for use after the iterator advances.
-```
-
-With `persist=False`, records are read-only proxies and cease to be valid after
-iteration advances. Copy needed strings or use the default when retaining
-records.
-
-### Quality scores
-
-```python
-with pysam.FastxFile("reads.fastq") as reads:
-    for record in reads:
-        if record.quality is None:  # FASTA record
-            continue
-        qualities = record.get_quality_array()
-        mean_quality = (
-            sum(qualities) / len(qualities) if qualities else None
-        )
-```
-
-Pysam converts the FASTQ quality string to numeric Phred scores. Confirm the
-source encoding for legacy FASTQ; modern data is normally Phred+33.
-
-### Safe writing
-
-`str(record)` preserves whether the record is FASTA or FASTQ and includes the
-comment and quality line correctly:
-
-```python
-with pysam.FastxFile("reads.fastq.gz") as source, open(
-    "filtered.fastq", "x", encoding="utf-8"
-) as destination:
-    for record in source:
-        qualities = record.get_quality_array()
-        if qualities and sum(qualities) / len(qualities) >= 20:
-            destination.write(str(record) + "\n")
-```
-
-Use exclusive creation (`"x"`) when replacement was not requested. Manual
-four-line FASTQ formatting can lose comments or create sequence/quality length
-mismatches.
-
-### Streaming statistics
-
-```python
-def fastx_stats(path: str) -> dict[str, float | int | None]:
-    record_count = 0
-    base_count = 0
-    quality_sum = 0
-    quality_count = 0
-
-    with pysam.FastxFile(path, persist=False) as records:
-        for record in records:
-            record_count += 1
-            base_count += len(record.sequence)
-            if record.quality is not None:
-                qualities = record.get_quality_array()
-                quality_sum += sum(qualities)
-                quality_count += len(qualities)
-
-    return {
-        "records": record_count,
-        "bases": base_count,
-        "mean_length": (
-            base_count / record_count if record_count else None
-        ),
-        "mean_quality": (
-            quality_sum / quality_count if quality_count else None
-        ),
-    }
-```
-
-This is a full scan but has constant memory.
-
-## BGZF and Tabix
-
-`TabixFile` provides random access to coordinate-sorted, BGZF-compressed
-tabular files.
-
-### Non-destructive compression and indexing
-
-```python
-import pysam
-
-pysam.tabix_compress("regions.bed", "regions.bed.gz")
-pysam.tabix_index("regions.bed.gz", preset="bed")
-```
-
-This leaves `regions.bed` intact. By contrast, calling `tabix_index()` directly
-on an uncompressed file may automatically compress it and remove the original
-unless `keep_original=True`.
-
-Do not use `force=True` unless replacing existing compressed data or indexes is
-explicitly intended.
-
-### Input requirements
-
-- Sort by contig and coordinate first. `tabix_index()` does not verify sort
-  order.
-- Use BGZF, not ordinary gzip.
-- Select the correct preset: commonly `bed`, `gff`, `sam`, or `vcf`.
-- Presets define columns and coordinate conventions.
-
-For a custom table:
-
-```python
-pysam.tabix_index(
-    "custom.tsv.gz",
-    seq_col=0,
-    start_col=1,
-    end_col=2,
-    zerobased=True,
-)
-```
-
-Python column indices are 0-based. File coordinates are assumed 1-based unless
-`zerobased=True`. This is separate from query coordinates, which are always
-numeric 0-based in the Python API.
-
-Use CSI for references beyond legacy TBI limits:
-
-```python
-pysam.tabix_index(
-    "regions.bed.gz",
-    preset="bed",
-    csi=True,
-    min_shift=14,
-)
-```
-
-### Querying
-
-```python
-with pysam.TabixFile(
-    "regions.bed.gz",
-    parser=pysam.asBed(),
-    threads=4,
-) as regions:
-    for row in regions.fetch("chr1", 1_000, 2_000):
-        print(row.contig, row.start, row.end, row.name)
-```
-
-Numeric query coordinates are 0-based, half-open. Region strings are
-samtools-style 1-based inclusive.
-
-Useful parsers:
-
-- `pysam.asTuple()`: tuple-like fields
-- `pysam.asBed()`: BED fields with 0-based start/end
-- `pysam.asGTF()`: GTF/GFF-like fields and attributes
-- `pysam.asVCF()`: lightweight tabix VCF parser
-
-For complete VCF semantics, use `VariantFile`, not `TabixFile(asVCF())`.
-
-Without a parser, each result is the raw tab-delimited string:
-
-```python
-with pysam.TabixFile("annotations.tsv.gz") as table:
-    for line in table.fetch("chr1", 1_000, 2_000):
-        fields = line.split("\t")
-```
-
-### Headers and multiple iterators
-
-```python
-with pysam.TabixFile("annotations.gff.gz") as table:
-    header_lines = list(table.header)
-    first = table.fetch("chr1", multiple_iterators=True)
-    second = table.fetch("chr2", multiple_iterators=True)
-```
-
-Header lines are yielded without trailing newlines. Each
-`multiple_iterators=True` iterator reopens the file and adds overhead.
-
-## Choosing the Right Interface
-
-| Data | Interface | Access |
-|---|---|---|
-| Reference FASTA | `FastaFile` | indexed random access |
-| FASTA/FASTQ reads | `FastxFile` | sequential |
-| BED/GFF/GTF/custom table | `TabixFile` | BGZF + tabix/CSI random access |
-| VCF/BCF | `VariantFile` | structured records, sequential or indexed |
-
-## Common Pitfalls
-
-- Mixing numeric 0-based coordinates with 1-based region strings
-- Expecting `FastaFile` to open without `.fai`
-- Using ordinary gzip for indexed FASTA or tabix data
-- Retaining a `persist=False` FASTX proxy
-- Assuming every FASTX record has qualities
-- Manually formatting FASTQ and losing comments or quality alignment
-- Letting `tabix_index()` remove the uncompressed source unexpectedly
-- Indexing an unsorted file
-- Forgetting `zerobased=True` for custom 0-based table coordinates
-- Using a TBI index for coordinates beyond its legacy range

package/skills/pysam/references/sources.md

@@ -1,132 +0,0 @@
-# Authoritative Sources
-
-Last researched: **2026-07-23**
-
-Skill baseline: **pysam 0.24.0**, released **2026-04-27**, wrapping
-**HTSlib/samtools/bcftools 1.23.1**.
-
-Use these sources in priority order when refreshing this skill. Do not assume
-that the newest standalone HTSlib release is the version embedded in the
-current pysam wheel.
-
-## Pysam Release and Package Metadata
-
-- [pysam on PyPI](https://pypi.org/project/pysam/) — current published version,
-  release date, wheels, release history, and provenance attestations
-- [Pysam release notes](https://pysam.readthedocs.io/en/latest/release.html) —
-  0.24 CRAM changes, Python support, bundled component versions, bug fixes, and
-  deprecations
-- [pysam-developers/pysam](https://github.com/pysam-developers/pysam) —
-  upstream source and current development state
-- [pysam 0.24.0 source tag](https://github.com/pysam-developers/pysam/tree/v0.24.0)
-  — version-specific implementation and tests
-
-As of the research date, PyPI identifies 0.24.0 as latest. The 0.24 release
-notes state:
-
-- tested Python versions: 3.8 through 3.14
-- wheels: macOS and Linux, ARM and x86-64
-- bundled HTSlib/samtools/bcftools: 1.23.1
-- default newly written CRAM: 3.1
-- implicit EBI CRAM reference fetching: removed
-- `format_options`: Python `str` values work as documented
-- top-level CIGAR constants remain compatibility aliases; prefer
-  `pysam.CIGAR_OPS`
-
-Re-check all of these before changing the version pin.
-
-## Official Pysam Documentation
-
-- [Documentation index](https://pysam.readthedocs.io/en/latest/index.html)
-- [Installation](https://pysam.readthedocs.io/en/latest/installation.html)
-- [Usage guide](https://pysam.readthedocs.io/en/latest/usage.html)
-- [API reference](https://pysam.readthedocs.io/en/latest/api.html)
-- [FAQ](https://pysam.readthedocs.io/en/latest/faq.html)
-- [Release notes](https://pysam.readthedocs.io/en/latest/release.html)
-- [Glossary](https://pysam.readthedocs.io/en/latest/glossary.html)
-
-Use the API reference for signatures and documented defaults. Use the FAQ for
-iterator lifetime, threading, coordinate, pileup, and quality-editing
-behavior. Where prose in the older usage guide conflicts with the current API,
-verify against the 0.24 source/tests and installed runtime.
-
-One known example: a real Python file object exposing `fileno()` works with
-`AlignmentFile`, while `io.BytesIO` does not. The API constructor documents
-file-object support; the usage guide's broad statement about "true python file
-objects" is too general.
-
-## Bundled Tool Manuals
-
-Pysam 0.24 wraps the 1.23.1 tool line. Consult manuals matching that line when
-using dispatchers:
-
-- [samtools 1.23 manual](https://www.htslib.org/doc/1.23/samtools.html)
-- [bcftools 1.23 manual](https://www.htslib.org/doc/1.23/bcftools.html)
-- [tabix 1.23 manual](https://www.htslib.org/doc/1.23/tabix.html)
-- [bgzip 1.23 manual](https://www.htslib.org/doc/1.23/bgzip.html)
-- [faidx 1.23 format/manual](https://www.htslib.org/doc/1.23/faidx.html)
-- [HTSlib documentation index](https://www.htslib.org/doc/)
-
-Pysam dispatchers emulate command-line subcommands but capture stdout/stderr.
-The pysam usage guide, not only the tool manual, defines `catch_stdout`,
-`save_stdout`, `split_lines`, `get_messages()`, and `SamtoolsError`.
-
-## CRAM References
-
-- [Pysam 0.24 release notes](https://pysam.readthedocs.io/en/latest/release.html#release-0-24-0)
-  — CRAM 3.1 default and removal of implicit EBI lookup
-- [Samtools reference-sequence guidance](https://www.htslib.org/doc/1.23/samtools.html)
-  — CRAM reference search order and `REF_PATH`/`REF_CACHE`
-- [Using CRAM within Samtools](https://www.htslib.org/workflow/cram.html) —
-  reference-based compression, M5 tags, local caches, and workflow concepts
-
-The older CRAM workflow page describes historical fallback to EBI. For pysam
-0.24/HTSlib 1.23 behavior, the 0.24 release notes and matching samtools manual
-take precedence: implicit EBI lookup is no longer the default.
-
-## Canonical Format Specifications
-
-- [HTS format specifications index](https://samtools.github.io/hts-specs/)
-- [SAM/BAM and BAI specification](https://samtools.github.io/hts-specs/SAMv1.pdf)
-- [SAM optional tags specification](https://samtools.github.io/hts-specs/SAMtags.pdf)
-- [CRAM 3 specification](https://samtools.github.io/hts-specs/CRAMv3.pdf)
-- [VCF 4.5 specification](https://samtools.github.io/hts-specs/VCFv4.5.pdf)
-- [BCF 2 quick reference](https://samtools.github.io/hts-specs/BCFv2_qref.pdf)
-- [Tabix index specification](https://samtools.github.io/hts-specs/tabix.pdf)
-- [CSI specification](https://samtools.github.io/hts-specs/CSIv1.pdf)
-- [BED 1 specification](https://samtools.github.io/hts-specs/BEDv1.pdf)
-
-Use format specifications for coordinate fields, flags, tags, header
-semantics, binary encodings, and index limits. Use pysam documentation for how
-those concepts are translated into Python properties.
-
-## Research Queries Used for This Refresh
-
-Parallel web search/extract was used for:
-
-- current stable pysam version, date, Python range, and wheel platforms
-- recent 0.22–0.24 release changes and deprecations
-- current AlignmentFile, pileup, VariantFile, FASTA/FASTQ, and Tabix APIs
-- current coordinate, iterator, thread-safety, and pileup FAQ guidance
-- HTSlib CRAM reference behavior and canonical format specifications
-
-Context7 cross-checked the upstream source documentation for:
-
-- pysam 0.24 release/CRAM behavior
-- AlignmentFile signatures and defaults
-- VariantFile, FastaFile, FastxFile, and TabixFile APIs
-
-No generated research JSON is part of this skill.
-
-## Refresh Checklist
-
-When updating:
-
-1. Check PyPI for the newest published pysam.
-2. Read all release notes since the pinned version.
-3. Record the embedded HTSlib/samtools/bcftools version.
-4. Verify supported Python versions and wheel platforms.
-5. Re-check CRAM defaults and reference lookup.
-6. Compare API signatures in `api.html` and the tagged source.
-7. Run bundled scripts against the new version.
-8. Re-run Agent Skills validation and the security scanner.