source-to-skill 0.7.0__tar.gz

source_to_skill-0.7.0/CHANGELOG.md

@@ -0,0 +1,379 @@
+# Changelog
+
+All notable changes to **source-to-skill** are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.7.0] - 2026-10-26
+
+Five new kinds of source, three companion skills, measured quality, and
+every major coding agent.
+
+### Added
+
+- **Documentation sites via llms.txt** — a `/llms.txt` or `/llms-full.txt`
+  URL, or any docs URL with `--type docs`, becomes a docs skill: the full
+  file split on its headings, or each linked page fetched in order
+  (Markdown preferred, HTML through the article extractor), with guides
+  that cite the live page.
+- **Notes folders** — a local directory (Obsidian vault, Markdown or
+  Logseq folder) becomes a skill of your own frameworks, hub notes and
+  recorded decisions: frontmatter tags kept, `[[wikilinks]]` and embeds
+  resolved, hubs ranked by inbound links, hidden folders skipped, and a
+  privacy rule in the template.
+- **Word, PowerPoint and OpenDocument** — `.docx .pptx .odt .odp` through
+  the standard library: heading or slide segments, tables, and speaker
+  notes kept with their slide. `.doc`, `.ppt` and `.rtf` get a re-save
+  hint.
+- **Hacker News and Reddit threads** — the post plus one segment per
+  top-level comment subtree in the site's own ranking; the template keeps
+  consensus, attributed tips and dissent apart.
+- **Meeting transcripts** — `.srt` and `.vtt` exports from Zoom, Teams or
+  Meet, with speaker labels; the template records decisions, action items
+  with owners (never invented), and who said what.
+- **Companion skills** in the plugin and the installer:
+  `skill-study` (one-question quizzes with Leitner spaced repetition,
+  explain-it-back checks, study plans, Anki export with deep links),
+  `skill-eval` (blind, source-grounded scoring of a generated skill) and
+  `skill-library` (inventory, search and refresh of every skill made).
+- **Grounded evals in every skill** — generation writes
+  `evals/evals.json` (skill-creator format) from the source while it is in
+  context; `validate_skill.py` checks it. Every gallery skill now carries
+  one plus a blind run: 127 of 128 expectations pass.
+- **Gallery: a podcast and a docs site** — `karpathy-agi-decade-away`
+  (Dwarkesh Patel with Andrej Karpathy, 2h25m, transcribed locally) and
+  `uv` (Astral's docs via llms.txt, 244k source tokens to a 1.7k
+  `SKILL.md`).
+- **Plugin eval suite** — `evals/` holds 12 `claude plugin eval` cases
+  that check the right skill loads for real requests (and stays out of
+  unrelated ones); a manual `Plugin evals` workflow runs them in CI.
+- **More agents** — `source-to-skill install --agent` now takes `codex`,
+  `gemini`, `cursor`, `opencode`, `kiro`, `windsurf` and `factory`, and
+  installs the companion skills next to `source-to-skill` (`--only` skips
+  them; `--dest PATH` installs only `source-to-skill`, into exactly that
+  folder). The README documents `npx skills add`.
+- `docs/llms.txt` for the project site.
+
+### Fixed
+
+- A direct podcast `.mp3` link that redirects to a signed CDN URL kept
+  the signed URL as `origin` and deep-link base, so links expired within
+  a day; the URL you passed is kept instead.
+- Articles behind an HTTP 308 redirect failed on Python 3.10, whose
+  urllib does not follow 308 (Anthropic's engineering posts moved that
+  way).
+- `--check` reported a Whisper backend that is installed but fails to
+  import (a native wheel built for another OS release) as missing, and
+  suggested installing it again; it now says `BROKEN` with the error.
+- Extraction no longer loses content silently: docs pages and notes left
+  out by the page, size or text caps are listed in the metadata
+  (`dropped`), a docs page too big to keep whole is skipped instead of
+  cut mid-text, and threads report `truncated`.
+- A note opening with a `---` horizontal rule lost everything up to the
+  next `---`; only YAML-looking blocks are treated as frontmatter.
+- A Reddit comment whose body is only whitespace entities crashed thread
+  extraction.
+- Transcript files kept only one of two identical consecutive lines
+  ("Yes." / "Yes."), and decoded non-UTF-8 files as Latin-1 instead of
+  Windows-1252.
+- A local media file with `#` or `?` in its name (`Meeting #4.mp4`) was
+  not recognised.
+- Repeated headings or slide titles in documents and llms-full.txt
+  collapsed into one segment in refresh diffs.
+- `install --force` could leave no skill behind if the final move
+  failed; the old copy is now restored.
+- `validate_skill.py` accepts no empty `evals` list, and the skill
+  library reads folded (`>`) and literal (`|`) descriptions.
+
+## [0.6.0] - 2026-10-05
+
+Whole podcast shows, real tables, and repo docs in reading order.
+
+### Added
+
+- **Podcast shows from RSS** — a feed URL (`feeds.*` hosts, `.rss`,
+  `.xml`, `/feed`, `/rss`) becomes a course skill of the latest 5
+  episodes (`--limit N`), each transcribed locally with Whisper and
+  titled from the feed (`"kind": "feed"`).
+- **Per-lesson deep links** — playlist segments carry a
+  `deep_link_template` (`&t=` for YouTube, `#t=` for media files), so
+  lessons from any source link moments, not just videos.
+- **Tables and images in books and articles** — HTML tables become
+  `| a | b |` rows with a header separator (layout-only tables stay plain
+  text); images with alt text become `[image: ...]` and captions
+  `[figure] ...` instead of stray lines.
+- **Article bylines** — author and date fall back to JSON-LD,
+  `twitter:creator`, and `rel="author"` links.
+- **Repo docs in reading order** — Sphinx `toctree` (from
+  `docs/index.rst`) or MkDocs `nav:` sets the segment order; `include` /
+  `literalinclude` (and MyST `{include}`) are resolved inside the repo, so
+  Flask's changelog and license are no longer empty stubs; reST roles
+  reduce to readable text, version and admonition directives to short
+  lines, autodoc to an "API reference:" line; segments flag `orphan` and
+  `stub`.
+- **Plugin validation in CI** — `claude plugin validate --strict` for the
+  marketplace and plugin manifests (the marketplace gained a
+  description).
+
+### Changed
+
+- Inline `[t=Ns]` markers every 30 s instead of 60 s: deep links land
+  within half a minute of a claim for about 3% more tokens.
+
+### Fixed
+
+- **Whisper fallback broke with yt-dlp plugins installed** — yt-dlp's
+  plugin loader can rebind `sys.modules["extractor"]` to its own package;
+  deferred imports then failed with "No module named
+  'extractor.media'". All extractor imports are now module-level.
+- **Podcast downloads printed a progress bar** into the extractor output;
+  downloads are now silent.
+
+## [0.5.0] - 2026-10-04
+
+Listen, not just watch and read: podcasts and audio become skills through
+local Whisper, YouTube channels become course skills, several sources merge
+into one topic skill, and skills can be refreshed. Plus a PyPI CLI, an
+examples gallery, CI, and a landing page.
+
+### Added
+
+- **Podcasts and audio** (`scripts/extractor/parsers/audio.py`) — Apple
+  Podcasts links, direct media URLs (`.mp3`, `.m4a`, `.wav`, `.mp4`, ...),
+  local audio/video files, and any page yt-dlp can download (Vimeo,
+  SoundCloud, TED, ... via `--type audio`) are transcribed locally with
+  Whisper. Segments follow the episode's chapters or 10-minute windows and
+  keep their start second; `deep_link_template` links back to the exact
+  moment where the origin is seekable (YouTube, Vimeo, direct media files).
+  Downloaded media is deleted after transcription.
+- **Local Whisper transcription** (`scripts/extractor/transcribe.py`) —
+  mlx-whisper (Apple Silicon, `whisper-large-v3-turbo`), faster-whisper,
+  or openai-whisper, first one installed wins; model override via
+  `SOURCE_TO_SKILL_WHISPER_MODEL`. No API keys.
+- **Captionless YouTube videos** — transcribed with Whisper instead of
+  failing when a backend is installed, in single videos and playlists;
+  `--transcribe` forces Whisper even when captions exist. Metadata
+  `captions` gains the value `whisper`.
+- **YouTube channels** — `youtube.com/@handle`, `/channel/...`, `/c/...`,
+  `/user/...` extract the latest 20 uploads as a course skill
+  (`"kind": "channel"`); `--limit N` changes that and caps long playlists.
+- **Topic skills from several sources** — `/source-to-skill <a> <b> <c>`
+  extracts each source into its own `--work-dir` and generates one skill
+  with source-tagged ideas, one file per source, and `disagreements.md`
+  recording where sources contradict instead of averaging them.
+- **Skill refresh** — every extraction writes `source.json` (origin,
+  options, segment keys), copied into the generated skill;
+  `/source-to-skill update <skill-dir>` re-extracts and
+  `tools/diff_source.py` reports what is new, so only new lessons are
+  generated.
+- **`--work-dir PATH`** — parallel or multi-source runs no longer share
+  one work directory.
+- **PyPI package and CLI** — `uvx source-to-skill install` (or `pipx`)
+  installs the skill for Claude Code, cross-agent, or Copilot CLI
+  (`--agent`, `--project`, `--dest`, `--force`); `extract`, `check`, and
+  `validate` wrap the bundled scripts. Extras: `youtube`, `paper`,
+  `audio`, `all`.
+- **Repo docs in reStructuredText and MDX** — `docs/` and `doc/` are read
+  for `*.md`, `*.mdx`, and `*.rst` (Sphinx projects such as Flask and
+  Django now extract their full docs); reST files are titled by their
+  first section title.
+- **Timestamps inside long chapters and lessons** — video, audio and
+  playlist transcripts carry an inline `[t=Ns]` marker about once a
+  minute, so skills can link a claim to its own second, not just the
+  chapter start.
+- **Article date fallback** — `<time datetime>` when no meta tag names
+  the date.
+- **`--check`** reports the Whisper backend and model, a missing ffmpeg,
+  and warns when the installed yt-dlp is over 60 days old.
+- **Examples gallery** (`examples/`) — skills generated by the tool itself
+  from a talk, a course playlist, a paper, a book, an article, a repo, and
+  a three-source topic; CI validates all of them.
+- **Project infrastructure** — CI (tests on Python 3.10–3.14 and macOS,
+  wheel install smoke test, example validation), a weekly live-source
+  smoke workflow, release workflow with PyPI trusted publishing, issue
+  forms, PR template, `SECURITY.md`, `CODE_OF_CONDUCT.md`,
+  `CITATION.cff`, Dependabot, and a GitHub Pages landing page.
+
+### Fixed
+
+- **Paper metadata and sections** — arXiv papers take title, authors, year
+  and DOI from the arXiv API (falling back to the PDF when it is
+  unreachable); local PDFs get their title from the largest page-1 font
+  instead of the file name. Years prefer the arXiv id or venue stamp over
+  dataset names like "WMT 2014". Numbered headings ("3 Model
+  Architecture", number on its own line), lettered appendices, and
+  tables of contents are recognized, so a paper no longer collapses into
+  one "Background" segment. References split into one entry per
+  reference (`[n]`, `[ADG+16]`, or numbered), with wrapped lines joined.
+  Metadata adds `arxiv_id` and `abstract_url`; `authors` is a list.
+- **JSON summary corrupted by library output** — stdout now carries only
+  the JSON summary; anything a library prints while parsing goes to
+  stderr, and PyMuPDF is imported as `pymupdf` (its legacy `fitz` name
+  prints a deprecation notice on new versions).
+- **Rate limits reported as missing captions** — a playlist whose videos
+  all hit YouTube's HTTP 429 now fails with "rate-limiting, wait and
+  retry" instead of suggesting Whisper, and stops after 3 rate-limited
+  videos in a row instead of requesting the rest. Single sources get the
+  same hint, and a YouTube bot check suggests updating yt-dlp.
+- **EPUB code lost its indentation** — `<pre>` blocks keep leading
+  whitespace (`git status -s` columns, YAML, Python), superscripts keep a
+  `^` (`2^80`, not `280`), and zero-width spaces are removed. Front-matter
+  segments (contents, contributors, dedications) are flagged
+  `front_matter` so generators skip them.
+- **No absolute home paths in `source.json`** — local sources are stored
+  as `~/...`.
+- **Repo segments name their file** — each segment has a `path`, so
+  same-titled pages (two "Templates") stay distinguishable.
+- **Caption notices in transcripts** — bracket-only cues (`[Music]`,
+  `[Submit subtitle corrections ...]`) are dropped.
+- **Article text glued at nested blocks** — a list nested right after an
+  inline label (`<li><b>Sectioning</b>:<ul>...`) no longer runs the label
+  into the next line's text.
+
+### Changed
+
+- The generator spec asks for an article byline and skips support files
+  that would only repeat a small source.
+
+## [0.4.0] - 2026-07-27
+
+The repo is now a Claude Code plugin marketplace, and the skill lives in a
+self-contained folder.
+
+### Added
+
+- **Plugin install** — `.claude-plugin/marketplace.json` +
+  `.claude-plugin/plugin.json`: install with
+  `/plugin marketplace add michalstrnadel/source-to-skill` and
+  `/plugin install source-to-skill@source-to-skill`.
+
+### Changed
+
+- **Breaking (install layout):** the skill moved from the repo root to the
+  self-contained `skills/source-to-skill/` folder (`SKILL.md`, `scripts/`,
+  `tools/` together). Existing installs that cloned the whole repo into
+  `~/.claude/skills/source-to-skill` must re-install: clone anywhere and
+  symlink/copy `skills/source-to-skill/` into the agent's skills folder
+  (see README Quick start).
+
+## [0.3.0] - 2026-07-27
+
+Three new source types: YouTube playlists become course skills, web
+articles become claims-and-highlights skills, GitHub repositories become
+library skills — articles and repos with zero new dependencies.
+
+### Added
+
+- **Playlist parser** (`scripts/extractor/parsers/playlist.py`) — YouTube
+  playlists via a yt-dlp flat listing plus per-video caption extraction
+  reusing the video parser. One segment per video with an extra `url` field;
+  videos without captions are skipped with a stderr warning and recorded in
+  `metadata["skipped"]` — one bad video never fails the playlist, which
+  errors only when no video has usable captions. Playlists over 30 videos
+  warn on stderr before proceeding. Metadata adds `channel`, `video_count`,
+  and `skipped` under `source_type: "playlist"`. A `watch?v=...&list=...`
+  URL stays a single video unless forced with `--type playlist`.
+- **Article parser** (`scripts/extractor/parsers/article.py`) — any other
+  `http(s)` URL becomes a web article, parsed with the standard library
+  alone: urllib fetch with a browser User-Agent, charset from the HTTP
+  header or meta tag, and an `html.parser` extractor that drops
+  script/style/nav/header/footer/aside/form subtrees, prefers `<article>`
+  over `<main>` over `<body>`, and splits segments at h1–h3 headings.
+  Title from `og:title` or `<title>`, best-effort `author` and `date`.
+  Pages yielding under 100 words fail with "Couldn't extract a readable
+  article" and a next step.
+- **Repo parser** (`scripts/extractor/parsers/repo.py`) — bare
+  `github.com/<owner>/<repo>` URLs download the codeload tarball (no git
+  clone) with path-traversal-safe extraction, then read the root README*,
+  top-level `*.md` (excluding LICENSE*), and `docs/**/*.md` in that order,
+  one segment per file titled by its first H1. Total text is capped at
+  ~2 MB with a warning listing dropped files. Missing or private repos
+  (404), other HTTP errors, and repos without any README or docs all fail
+  with clear errors. Metadata adds `owner`, `repo`, and `file_count` under
+  `source_type: "repo"`.
+- **Detection order** (`scripts/extractor/utils.py`) — YouTube video →
+  playlist → arXiv → GitHub repo → generic http(s) article → `.pdf` →
+  `.epub`; `--type` validation extended (`playlist` valid on
+  `watch?list=` URLs, `article` on any http URL, `repo` only on GitHub
+  repo URLs).
+- **Skill templates** (repo-root `SKILL.md`, Step 5) — playlist/course
+  skills get a course overview, a lesson index table, per-video
+  `lessons/NN-<slug>.md` files opening with the video URL, and a
+  course-wide cheatsheet; article skills get thesis plus key claims with a
+  link to the original and `highlights.md` quotes; repo skills get what
+  the project does, install, core usage patterns, a doc index,
+  `guides/*.md` per doc area, and a commands-and-snippets cheatsheet.
+- **Docs** — README gains a Supported sources table, quick-start examples,
+  features, and troubleshooting entries for the three new types;
+  architecture docs cover the new parsers and detection order.
+
+## [0.2.0] - 2026-07-27
+
+Books join videos and papers: EPUBs with zero new dependencies, PDF books
+via a type override.
+
+### Added
+
+- **Book parser** (`scripts/extractor/parsers/book.py`) — EPUB books parsed
+  with the Python standard library alone (zip → `META-INF/container.xml` →
+  OPF spine; chapter titles from `toc.ncx` or the EPUB3 `nav.xhtml`, falling
+  back to the first `<h1>`, the document `<title>`, then "Chapter N"; tags
+  stripped and entities decoded via `html.parser`). PDF books reuse PyMuPDF:
+  chapters come from the top-level PDF outline, falling back to section
+  detection. Metadata uses `source_type: "book"` and adds `author` (EPUB
+  `dc:creator` / PDF metadata). Malformed EPUBs (bad zip, missing
+  container.xml, unreadable OPF, empty spine) fail with clear errors; a
+  malformed `toc.ncx` falls back to nav/heading titles instead of aborting,
+  spine items missing from the archive are skipped with a stderr warning,
+  and unresolved PDF outline destinations are ignored.
+- **`--type` override** — `extract.py <source> [--type youtube|paper|book]`
+  forces the parser when auto-detection is not enough (a PDF book vs. a PDF
+  paper); `.epub` auto-detects as book, unknown types and impossible
+  source/type combinations are rejected with clear errors.
+- **Book skill template** (repo-root `SKILL.md`, Step 5) — core mental
+  models plus a chapter index table in the generated SKILL.md, on-demand
+  `chapters/NN-<slug>.md` files, `glossary.md` with chapter references, and
+  `cheatsheet.md` with decision rules and anti-patterns.
+
+## [0.1.0] - 2026-07-27
+
+First public release: YouTube videos and academic papers in, installable
+agent skills out.
+
+### Added
+
+- **YouTube parser** (`scripts/extractor/parsers/youtube.py`) — captions via
+  yt-dlp (manual subtitles preferred, auto-generated as fallback; English
+  variants preferred), VTT parsing with auto-caption de-duplication, and
+  segmentation by the video's own chapters with fixed 10-minute windows as
+  fallback. Every segment records its start timestamp so generated skills
+  can deep-link back into the video with `&t=`.
+- **Paper parser** (`scripts/extractor/parsers/paper.py`) — text extraction
+  via PyMuPDF from local PDFs or modern arXiv URLs (downloaded
+  automatically), heuristic section detection (Abstract through References,
+  with a safe full-text fallback), DOI/year detection, per-section page
+  numbers, and a raw reference list. Scanned PDFs without a text layer fail
+  with a clear "OCR not supported yet" error.
+- **Extractor CLI** (`scripts/extract.py`) — detects the source type,
+  dispatches to the right parser, and writes `full_text.txt` +
+  `metadata.json` (the shared segment contract: `title`, `start_s`, `pages`,
+  `offset`) to the work dir. `--check` reports each optional dependency
+  (yt-dlp, PyMuPDF) with an exact install hint; extraction errors state what
+  failed and the next step.
+- **Generator spec** (repo-root `SKILL.md`) — the six-step agent workflow:
+  extract, confirm token cost, analyze structure (segment-by-segment reads
+  for large sources), choose an install target, generate, verify. Video
+  skills get a timestamped segment index, per-segment files opening with a
+  deep link, and a cheatsheet; paper skills get a TL;DR plus
+  `methods.md`, `findings.md`, `limitations.md`, `glossary.md`, and
+  `citations.md`.
+- **Skill validator** (`tools/validate_skill.py`) — checks generated skills
+  for valid frontmatter (name matches the directory, description within
+  limits), resolvable markdown links, and no empty files; prints
+  `OK: skill is valid` on success.
+- **Offline test suite** — pytest coverage for source detection, VTT
+  parsing, segmentation, paper section detection, dependency probing, the
+  CLI, and the validator; yt-dlp is mocked and the PDF fixture is generated
+  locally, so no test touches the network.
+- MIT license; Python ≥ 3.10.

source_to_skill-0.7.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Michal Strnadel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

source_to_skill-0.7.0/MANIFEST.in

@@ -0,0 +1,4 @@
+# sdist only (the wheel's contents come from [tool.setuptools] in pyproject.toml)
+include CHANGELOG.md
+graft tests
+global-exclude *.py[cod] .DS_Store