distillr 0.1.0__tar.gz

distillr-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nick Seal
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

distillr-0.1.0/PKG-INFO

@@ -0,0 +1,257 @@
+Metadata-Version: 2.4
+Name: distillr
+Version: 0.1.0
+Summary: Source-to-intelligence platform: turn YouTube, websites, and arXiv papers into a structured, reusable corpus with per-source insights, cross-source synthesis, and Deep Research reports.
+Author: Nick Seal
+License: MIT
+Project-URL: Homepage, https://github.com/blisspixel/distillr
+Project-URL: Repository, https://github.com/blisspixel/distillr
+Project-URL: Issues, https://github.com/blisspixel/distillr/issues
+Keywords: research,intelligence,synthesis,arxiv,youtube,rag,knowledge-base,mcp
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: yt-dlp>=2025.1.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: google-genai>=1.50.0
+Requires-Dist: typer>=0.9.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pydantic-settings>=2.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: python-docx>=1.0.0
+Requires-Dist: playwright>=1.52.0
+Requires-Dist: mcp>=1.0.0
+Requires-Dist: fastapi>=0.115.0
+Requires-Dist: uvicorn[standard]>=0.30.0
+Requires-Dist: jinja2>=3.1.0
+Requires-Dist: markdown>=3.5.0
+Requires-Dist: pypdf>=4.0.0
+Requires-Dist: requests>=2.32.0
+Requires-Dist: defusedxml>=0.7.1
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: ruff>=0.6.0; extra == "dev"
+Requires-Dist: bandit[toml]>=1.7; extra == "dev"
+Requires-Dist: pip-audit>=2.7; extra == "dev"
+Requires-Dist: pre-commit>=3.5; extra == "dev"
+Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+# Distill
+
+[![CI](https://github.com/blisspixel/distillr/actions/workflows/ci.yml/badge.svg)](https://github.com/blisspixel/distillr/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/distillr.svg)](https://pypi.org/project/distillr/)
+[![Python](https://img.shields.io/pypi/pyversions/distillr.svg)](https://pypi.org/project/distillr/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
+
+> Turn YouTube, websites, and arXiv papers into a structured, reusable corpus of insights, syntheses, and reports — all plain markdown on your disk.
+
+```bash
+pip install distillr
+distill papers "temporal knowledge graph" --topic tkg --limit 20
+```
+
+That one command searches arXiv, downloads 20 PDFs, extracts full text, runs structured analysis on each, and writes a cross-paper synthesis. For a 20-paper run like the example below, expect single-digit minutes and roughly ~$1 in model spend. Terminal output during the run looks like this:
+
+```
+Papers: temporal knowledge graph
+Topic: tkg | Selected papers: 20
+
+  [1/20] Time is Not a Label: Continuous Phase Rotation for Temporal Knowledge
+         Graphs and Agentic Memory
+  [2/20] Inductive Reasoning for Temporal Knowledge Graphs with Emerging Entities
+  ...
+
+  6m 47s  ~$1.01 (391,278 in / 38,117 out)
+
+  paper.md     90.4 KB
+  insights.md   8.1 KB
+  ...
+  paper_synthesis.md  11.8 KB
+  corpus_synthesis.md 10.5 KB
+```
+
+## What you get
+
+One local `library/` directory of plain markdown. No database, no cloud lock-in, no proprietary format. Open it in any text editor, Obsidian, VS Code, or feed it into another tool.
+
+Three source types, same pipeline shape (capture → analyze → synthesize → report):
+
+- **YouTube** — channels, topic searches, videos, Shorts
+- **Websites** — vendor sites, research hubs, curated URL sets (browser-first crawl with PDF/embedded-video ingestion)
+- **arXiv papers** — phrase-matched search, full-PDF extraction, structured per-paper insights, cross-paper synthesis
+
+Plus an MCP server so AI assistants and agent systems can query the library directly.
+
+## Quick start
+
+```bash
+pip install distillr
+playwright install chromium     # for YouTube search + website capture
+distill doctor                  # verify API keys + system health
+```
+
+Set two keys in `.env` (copy from `.env.example`):
+
+```bash
+XAI_API_KEY=xai-...             # Grok models
+GEMINI_API_KEY=AIza...          # Gemini Deep Research (reports + briefings)
+```
+
+Then try any of:
+
+```bash
+# Goal-aware cross-source discovery (papers + videos, reranked against a goal)
+distill discover "help an AI become a great music composer" --topic music --preview
+distill discover --goal-file private/my-goal.md --topic research --yes
+
+# Get smart on a YouTube topic, fast
+distill latest "Microsoft Fabric best practices" --limit 10 --report
+
+# Discover and ingest arXiv papers — expands the query, LLM-reranks candidates,
+# picks the top N (use --preview to see the shortlist without ingesting)
+distill papers "agent memory systems" --topic memory --limit 20
+distill papers "agent memory systems" --topic memory --limit 20 --preview
+
+# Distill a vendor/research site
+distill site-batch configs/example_seeds.json --topic example --seed-only
+```
+
+The full command reference lives in [`docs/usage.md`](docs/usage.md).
+
+## Mental model
+
+```
+library/
+  └── topics/<topic>/
+       ├── channels/<creator>/videos/<video>/
+       │     ├── transcript.txt
+       │     └── insights.md
+       ├── sites/<hostname>/pages/<page>/
+       │     ├── content.md
+       │     └── insights.md
+       ├── papers/<paper>/
+       │     ├── paper.md
+       │     └── insights.md
+       ├── topic_synthesis.md      # cross-source
+       └── corpus_synthesis.md     # mixed-source view
+```
+
+You build a topic library over time. Ingest once, refresh on a cadence, generate a report or briefing when you need one.
+
+See [`docs/outputs.md`](docs/outputs.md) for what every artifact contains.
+
+## Sample output
+
+A per-paper `insights.md` (excerpt):
+
+```markdown
+---
+paper_title: "Time is Not a Label: Continuous Phase Rotation for Temporal Knowledge Graphs"
+paper_id: 2604.11544v1
+analyzed_by: grok-4.20-0309-reasoning
+source_mode: full_pdf
+---
+
+### Core Contribution
+1. Continuous functional rotation θ_r(τ) = s · α_r · τ · ω instead of discrete
+   timestamp lookup tables. Zero-shot interpolation of unseen dates.
+2. Semantic Speed Gate: MLP that reads only text embedding ϕ(r) and outputs α_r.
+   Learns relational volatility from data.
+3. Geometric shadowing in complex space: obsolete facts rotated out of phase so
+   the correct fact outranks contradictions via the scoring function alone.
+
+### Methods and Evidence
+- On ICEWS05-15, RoMem-ChronoR reaches 72.6 MRR (vs vanilla ChronoR 68.4).
+- Zero-shot domain transfer to FinTMMBench: 0.728 MRR, 0.673 R@5.
+- All baselines use identical answer LLM and judge for fairness.
+
+### Limits and Open Questions
+- Computational cost at millions-of-facts scale is motivation but no latency,
+  memory, or throughput numbers are reported.
+- Gate pretrained only on ICEWS05-15 political events; generalization to
+  highly ambiguous relations is not quantified.
+```
+
+A cross-paper `paper_synthesis.md` (excerpt):
+
+```markdown
+## Strongest Research Signals
+
+- Append-only temporal representations improve long-horizon extrapolation:
+  RoMem (arXiv:2604.11544), EST (arXiv:2602.12389v3), and CID-TKG converge on
+  persistent or dual-view entity state over destructive overwriting, with
+  consistent MRR/Hits@K gains on ICEWS and GDELT.
+
+- Semantic gating scales better than manual relation tagging: RoMem's Semantic
+  Speed Gate and EST's energy-barrier gate both learn relational volatility
+  from text embeddings rather than schema tags…
+```
+
+For **multi-topic** literature reviews, stakeholder briefings, or agent grounding, `distill research-brief` (Gemini Deep Research, web-augmented) and `distill synthesize` (Grok 4.20 single-call, corpus-only) take a user-written context file that shapes the output. See [`docs/usage.md#research-briefings-and-deep-synthesis`](docs/usage.md#research-briefings-and-deep-synthesis).
+
+## Dashboard
+
+```bash
+distill                         # terminal home screen
+distill serve                   # local web dashboard at http://127.0.0.1:8899
+```
+
+The terminal home screen shows tracked topics, channel and topic watches, recent runs, failures, and rolling spend. The web dashboard adds clickable drill-downs to per-topic, per-channel, and per-video views with rendered markdown, plus cost history and watchlist status. Both auto-refresh and read directly from library files — no database.
+
+## MCP server
+
+Claude Desktop / Claude Code config:
+
+```json
+{ "mcpServers": { "distill": { "command": "distill-mcp" } } }
+```
+
+Distill exposes 8 tools, 12 resources, and 4 prompts. See [`docs/mcp.md`](docs/mcp.md) for the list.
+
+## Cost
+
+Bulk video analysis is essentially free (~$0.006/video). Gemini Deep Research dominates paid reports (~$2–3/report). `distill synthesize` is ~$0.50 for a multi-topic corpus pass. Every run logs actual vs estimated cost to `library/cost_log.jsonl`; `distill costs` shows the history.
+
+Full cost model in [`docs/cost.md`](docs/cost.md).
+
+## Docs
+
+- [`docs/usage.md`](docs/usage.md) — full command reference
+- [`docs/architecture.md`](docs/architecture.md) — data flow, 4-phase report pipeline, model routing, security hardening
+- [`docs/outputs.md`](docs/outputs.md) — what every artifact contains
+- [`docs/cost.md`](docs/cost.md) — cost model, examples, guardrails
+- [`docs/mcp.md`](docs/mcp.md) — MCP tools, resources, prompts
+- [`docs/briefing-contexts/TEMPLATE.md`](docs/briefing-contexts/TEMPLATE.md) — starting point for `--context-file` prompts
+- [`private/README.md`](private/README.md) — where personal/client-specific files go (git-ignored)
+
+## Roadmap and changelog
+
+- [`docs/CHANGELOG.md`](docs/CHANGELOG.md) — what shipped in `0.1.0`
+- [`ROADMAP.md`](ROADMAP.md) — what's next
+
+## Contributing
+
+See [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) for dev setup, quality gates, and scope. Security disclosures go through [`docs/SECURITY.md`](docs/SECURITY.md).
+
+## License
+
+MIT — see [`LICENSE`](LICENSE).

distillr-0.1.0/README.md

@@ -0,0 +1,202 @@
+# Distill
+
+[![CI](https://github.com/blisspixel/distillr/actions/workflows/ci.yml/badge.svg)](https://github.com/blisspixel/distillr/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/distillr.svg)](https://pypi.org/project/distillr/)
+[![Python](https://img.shields.io/pypi/pyversions/distillr.svg)](https://pypi.org/project/distillr/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
+
+> Turn YouTube, websites, and arXiv papers into a structured, reusable corpus of insights, syntheses, and reports — all plain markdown on your disk.
+
+```bash
+pip install distillr
+distill papers "temporal knowledge graph" --topic tkg --limit 20
+```
+
+That one command searches arXiv, downloads 20 PDFs, extracts full text, runs structured analysis on each, and writes a cross-paper synthesis. For a 20-paper run like the example below, expect single-digit minutes and roughly ~$1 in model spend. Terminal output during the run looks like this:
+
+```
+Papers: temporal knowledge graph
+Topic: tkg | Selected papers: 20
+
+  [1/20] Time is Not a Label: Continuous Phase Rotation for Temporal Knowledge
+         Graphs and Agentic Memory
+  [2/20] Inductive Reasoning for Temporal Knowledge Graphs with Emerging Entities
+  ...
+
+  6m 47s  ~$1.01 (391,278 in / 38,117 out)
+
+  paper.md     90.4 KB
+  insights.md   8.1 KB
+  ...
+  paper_synthesis.md  11.8 KB
+  corpus_synthesis.md 10.5 KB
+```
+
+## What you get
+
+One local `library/` directory of plain markdown. No database, no cloud lock-in, no proprietary format. Open it in any text editor, Obsidian, VS Code, or feed it into another tool.
+
+Three source types, same pipeline shape (capture → analyze → synthesize → report):
+
+- **YouTube** — channels, topic searches, videos, Shorts
+- **Websites** — vendor sites, research hubs, curated URL sets (browser-first crawl with PDF/embedded-video ingestion)
+- **arXiv papers** — phrase-matched search, full-PDF extraction, structured per-paper insights, cross-paper synthesis
+
+Plus an MCP server so AI assistants and agent systems can query the library directly.
+
+## Quick start
+
+```bash
+pip install distillr
+playwright install chromium     # for YouTube search + website capture
+distill doctor                  # verify API keys + system health
+```
+
+Set two keys in `.env` (copy from `.env.example`):
+
+```bash
+XAI_API_KEY=xai-...             # Grok models
+GEMINI_API_KEY=AIza...          # Gemini Deep Research (reports + briefings)
+```
+
+Then try any of:
+
+```bash
+# Goal-aware cross-source discovery (papers + videos, reranked against a goal)
+distill discover "help an AI become a great music composer" --topic music --preview
+distill discover --goal-file private/my-goal.md --topic research --yes
+
+# Get smart on a YouTube topic, fast
+distill latest "Microsoft Fabric best practices" --limit 10 --report
+
+# Discover and ingest arXiv papers — expands the query, LLM-reranks candidates,
+# picks the top N (use --preview to see the shortlist without ingesting)
+distill papers "agent memory systems" --topic memory --limit 20
+distill papers "agent memory systems" --topic memory --limit 20 --preview
+
+# Distill a vendor/research site
+distill site-batch configs/example_seeds.json --topic example --seed-only
+```
+
+The full command reference lives in [`docs/usage.md`](docs/usage.md).
+
+## Mental model
+
+```
+library/
+  └── topics/<topic>/
+       ├── channels/<creator>/videos/<video>/
+       │     ├── transcript.txt
+       │     └── insights.md
+       ├── sites/<hostname>/pages/<page>/
+       │     ├── content.md
+       │     └── insights.md
+       ├── papers/<paper>/
+       │     ├── paper.md
+       │     └── insights.md
+       ├── topic_synthesis.md      # cross-source
+       └── corpus_synthesis.md     # mixed-source view
+```
+
+You build a topic library over time. Ingest once, refresh on a cadence, generate a report or briefing when you need one.
+
+See [`docs/outputs.md`](docs/outputs.md) for what every artifact contains.
+
+## Sample output
+
+A per-paper `insights.md` (excerpt):
+
+```markdown
+---
+paper_title: "Time is Not a Label: Continuous Phase Rotation for Temporal Knowledge Graphs"
+paper_id: 2604.11544v1
+analyzed_by: grok-4.20-0309-reasoning
+source_mode: full_pdf
+---
+
+### Core Contribution
+1. Continuous functional rotation θ_r(τ) = s · α_r · τ · ω instead of discrete
+   timestamp lookup tables. Zero-shot interpolation of unseen dates.
+2. Semantic Speed Gate: MLP that reads only text embedding ϕ(r) and outputs α_r.
+   Learns relational volatility from data.
+3. Geometric shadowing in complex space: obsolete facts rotated out of phase so
+   the correct fact outranks contradictions via the scoring function alone.
+
+### Methods and Evidence
+- On ICEWS05-15, RoMem-ChronoR reaches 72.6 MRR (vs vanilla ChronoR 68.4).
+- Zero-shot domain transfer to FinTMMBench: 0.728 MRR, 0.673 R@5.
+- All baselines use identical answer LLM and judge for fairness.
+
+### Limits and Open Questions
+- Computational cost at millions-of-facts scale is motivation but no latency,
+  memory, or throughput numbers are reported.
+- Gate pretrained only on ICEWS05-15 political events; generalization to
+  highly ambiguous relations is not quantified.
+```
+
+A cross-paper `paper_synthesis.md` (excerpt):
+
+```markdown
+## Strongest Research Signals
+
+- Append-only temporal representations improve long-horizon extrapolation:
+  RoMem (arXiv:2604.11544), EST (arXiv:2602.12389v3), and CID-TKG converge on
+  persistent or dual-view entity state over destructive overwriting, with
+  consistent MRR/Hits@K gains on ICEWS and GDELT.
+
+- Semantic gating scales better than manual relation tagging: RoMem's Semantic
+  Speed Gate and EST's energy-barrier gate both learn relational volatility
+  from text embeddings rather than schema tags…
+```
+
+For **multi-topic** literature reviews, stakeholder briefings, or agent grounding, `distill research-brief` (Gemini Deep Research, web-augmented) and `distill synthesize` (Grok 4.20 single-call, corpus-only) take a user-written context file that shapes the output. See [`docs/usage.md#research-briefings-and-deep-synthesis`](docs/usage.md#research-briefings-and-deep-synthesis).
+
+## Dashboard
+
+```bash
+distill                         # terminal home screen
+distill serve                   # local web dashboard at http://127.0.0.1:8899
+```
+
+The terminal home screen shows tracked topics, channel and topic watches, recent runs, failures, and rolling spend. The web dashboard adds clickable drill-downs to per-topic, per-channel, and per-video views with rendered markdown, plus cost history and watchlist status. Both auto-refresh and read directly from library files — no database.
+
+## MCP server
+
+Claude Desktop / Claude Code config:
+
+```json
+{ "mcpServers": { "distill": { "command": "distill-mcp" } } }
+```
+
+Distill exposes 8 tools, 12 resources, and 4 prompts. See [`docs/mcp.md`](docs/mcp.md) for the list.
+
+## Cost
+
+Bulk video analysis is essentially free (~$0.006/video). Gemini Deep Research dominates paid reports (~$2–3/report). `distill synthesize` is ~$0.50 for a multi-topic corpus pass. Every run logs actual vs estimated cost to `library/cost_log.jsonl`; `distill costs` shows the history.
+
+Full cost model in [`docs/cost.md`](docs/cost.md).
+
+## Docs
+
+- [`docs/usage.md`](docs/usage.md) — full command reference
+- [`docs/architecture.md`](docs/architecture.md) — data flow, 4-phase report pipeline, model routing, security hardening
+- [`docs/outputs.md`](docs/outputs.md) — what every artifact contains
+- [`docs/cost.md`](docs/cost.md) — cost model, examples, guardrails
+- [`docs/mcp.md`](docs/mcp.md) — MCP tools, resources, prompts
+- [`docs/briefing-contexts/TEMPLATE.md`](docs/briefing-contexts/TEMPLATE.md) — starting point for `--context-file` prompts
+- [`private/README.md`](private/README.md) — where personal/client-specific files go (git-ignored)
+
+## Roadmap and changelog
+
+- [`docs/CHANGELOG.md`](docs/CHANGELOG.md) — what shipped in `0.1.0`
+- [`ROADMAP.md`](ROADMAP.md) — what's next
+
+## Contributing
+
+See [`docs/CONTRIBUTING.md`](docs/CONTRIBUTING.md) for dev setup, quality gates, and scope. Security disclosures go through [`docs/SECURITY.md`](docs/SECURITY.md).
+
+## License
+
+MIT — see [`LICENSE`](LICENSE).

distillr-0.1.0/distill/__init__.py

@@ -0,0 +1 @@
+"""Distill — Turn YouTube channels into strategic intelligence."""