khiip 0.2.0__tar.gz

khiip-0.2.0/.gitignore

@@ -0,0 +1,39 @@
+# Secrets / credentials
+.env
+.env.*
+*.key
+*.pem
+.aws/
+.config/khiip/
+
+# OS / editor
+.DS_Store
+.vscode/
+.idea/
+*.swp
+
+# Python
+__pycache__/
+*.pyc
+*.egg-info/
+.venv/
+venv/
+uv.lock
+dist/
+build/
+.pytest_cache/
+.ruff_cache/
+.python-version
+
+# JS (plugin scaffolding when added)
+node_modules/
+
+# Runtime data (daemon writes locally)
+*.sqlite
+*.db
+*.log
+
+# Local-only working files — never committed to the public repo.
+/CLAUDE.md
+/_internal/
+/khiip-internal/

khiip-0.2.0/CHANGELOG.md

@@ -0,0 +1,489 @@
+# Changelog
+
+All notable changes to **Khiip** are documented in this file. Format adapted
+from [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versioning per
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html). v0.x is pre-1.0;
+substrate decisions can still change until v1.0 stabilizes the contract.
+
+## [0.2.0] — 2026-06-26
+
+### Packaging
+
+- **On PyPI.** `uv tool install khiip` / `pipx install khiip` install the daemon,
+  CLI, and MCP server directly — no clone required.
+- **Open-core packaging.** The free AGPL core ships as `khiip`; the paid Khiip Plus
+  tier ships as a separate distribution. The published `khiip` sdist + wheel are
+  built from an allowlist and carry the core package only.
+- Release automation: tag-driven build → carve guard → PyPI Trusted Publishing
+  (OIDC, no API token in the repo). CI now runs the hermetic test suite on every PR.
+
+## [0.1.8] — 2026-06-14
+
+Captures can now be **re-derived offline from preserved bytes** — improve an
+extractor or a renderer and every old capture upgrades, without re-fetching, and
+even after the source post is deleted. This is the operational core of
+capture-and-keep: the bytes are the durable asset; the typed payload and every
+rendering are recomputable views over them. This release also deepens extraction
+across every in-band source and gives each source a platform-faithful card in
+Obsidian.
+
+### Added — substrate
+
+- **`refetch --re-extract` (offline L1→L2 replay).** Re-derives the typed
+  payload from the **preserved source bytes** with the *current* extractor — no
+  network — recovering fields an older extractor missed and surviving source
+  deletion. **In place**: same capture id / vault path / `valid_from` (re-parsing
+  unchanged bytes is a derived-cache refresh, not a new observation — only the
+  network `extraction` dimension supersedes). Re-renders the body and re-embeds
+  for recall. New optional `ByteReExtractor` protocol method `extract_from_bytes`
+  implemented by all six extractors (web / reddit / x / wiki / youtube / pdf);
+  daemon dimension `re-extract`, CLI `--re-extract`, MCP `dimension="re-extract"`.
+  Returns a clear 400 when no bytes were preserved (use `extraction`). This
+  is the replayability property of the preserved Source-tier bytes.
+- **`refetch --re-render` (offline L2→md replay).** Re-renders the markdown body
+  from the existing typed payload — apply an improved renderer or skin to a
+  capture (or the whole corpus) with no network, no re-extract, no re-embed. In
+  place. Daemon dimension `re-render`, CLI `--re-render`, MCP. Also a tight
+  developer loop for iterating renderers without re-fetching.
+- **Richer extraction (Tier-1 completeness lifts).** Fields already fetched but
+  never lifted into the typed payload now land in L2 (so MCP/REST agents, recall,
+  and renders can use them): YouTube `tags` / `categories` / `video_id` /
+  `channel_id` / `channel_url`; Web article `author` (byline); Wikipedia
+  `last_modified` / `last_rev_id`. Wikipedia also now fetches and stores
+  `categories` (`prop=categories`) at capture. Renderers surface these where
+  platform-faithful (YouTube channel link, Web byline, Wikipedia categories +
+  "last edited" footer). Existing captures gain them via `--re-extract`.
+- **Tier-2 deeper extraction.** More of each page is now typed into L2:
+  - **X** — author verification (`is_verified` / `verified_type` =
+    individual/business/government), typed polls (`Poll` / `PollChoice`),
+    and note-tweet inline bold; closed the vxtwitter media `alt_text` gap.
+  - **Reddit** — superscript + spoiler richtext preserved (`^(…)` / `>!…!<`)
+    instead of being flattened by the markdown converter.
+  - **Web** — `date_modified` / `date_published` / `publisher` / `breadcrumbs`
+    (JSON-LD with OG fallback), plus per-figure **caption + photo credit** (the
+    `<figure>`/`<figcaption>` standard + common credit patterns).
+- **Web inline media recovery.** Inline article **images** and embedded **video**
+  that the markdown converter previously dropped are now preserved: trafilatura's
+  inline images render in document position with their caption/credit, and video
+  embeds (YouTube / Vimeo / Dailymotion / HTML5 `<video>`) are kept as a typed
+  `Media` entry plus a body link. Every figure is lifted into `WebPayload.media`
+  so agents/recall see the full image set. New `WebPayload.site_icon_url` (the
+  site favicon). All retroactive via `--re-extract`.
+- **YouTube transcript export.** `GET /api/v1/captures/{id}?format=srt|vtt`
+  returns the stored transcript as SubRip or WebVTT — pure local conversion over
+  existing cues, no network; 404 for non-YouTube captures / missing transcripts.
+
+### Added — surfaces / renderers
+
+- **Per-source platform-faithful cards.** X, Reddit, YouTube, Wikipedia, and Web
+  captures each render in a per-source coloured card with platform-appropriate
+  metadata (X status footer + engagement glyphs + polls; Reddit flair + vote
+  pills + comment tree; YouTube channel/stats). The typed payload stays clean —
+  the styling is a *derived* Obsidian view (the plugin ships the CSS); in any
+  other markdown viewer the card degrades to standard markdown + content.
+- **Web news-masthead.** Web captures now read like a news-article header: a
+  `Web · <Type>` banner (News Article / Blog / Review / How-To / Recipe, from
+  schema.org), a brand band carrying the site **favicon + name + section**, the
+  headline, and an `By <author> · published · updated` byline; the description
+  becomes the standfirst dek. Tags render as individual hashtags.
+
+### Fixed
+
+- **Superseded marker preserved on in-place rewrites.** Re-extract / re-render /
+  media / wayback refreshes no longer strip the `superseded_by` banner +
+  frontmatter from an older note (it's display-only state; the DB was always
+  correct).
+- **Web inline links no longer scrambled.** The article body is now serialized
+  via trafilatura HTML → markdownify, fixing displaced/duplicated link text that
+  trafilatura's own markdown serializer produced on link-dense articles.
+- **Reddit admin/mod badges detected.** Distinguished status is read from the
+  author anchor (`<a class="author admin">`) rather than the comment container,
+  so admin/moderator comments are now flagged (previously none were). Backfills
+  via `--re-extract`.
+- **Comma-joined tags split correctly.** A `keywords` / `article:tag` value
+  emitted as a single `"a,b,c"` string (e.g. The Guardian) now splits into
+  separate hashtags instead of collapsing into one long underscore chain.
+
+## [0.1.7] — 2026-06-10
+
+Captures now carry a per-source identity banner so each source is recognisable
+at a glance in Obsidian.
+
+### Added — substrate
+
+- **Per-source identity banner.** Each capture's rendered body opens with a
+  `> [!khiip-<source>]` callout carrying a label + identifier (e.g.
+  `𝕏 · @jack`, `Reddit · r/selfhosted`, `YouTube · 3Blue1Brown`). It's plain
+  Markdown — in any viewer it degrades to an ordinary titled callout — so the
+  substrate stays destination-agnostic; the *visual* per-source colour + icon
+  is supplied by the consumer (the Obsidian plugin ships CSS targeting
+  `.callout[data-callout="khiip-<source>"]`). New `renderers/source_banner.py`
+  (`compose_source_banner`); prepended in the capture + supersede write paths.
+  This is the lightweight in-vault version of the deferred Phase 2.5+
+  source-styled renderer.
+
+## [0.1.6] — 2026-06-10
+
+Refetch versioning is now fork-safe, and recall returns one hit per URL. A live
+Obsidian-plugin smoke surfaced that repeatedly re-fetching the same URL (e.g. a
+"refresh" button, or an MCP agent holding a stale capture id) forked the
+supersession chain into multiple "current" captures for one URL.
+
+### Fixed — substrate
+
+- **`refetch` (extraction dimension) now resolves to the chain head before
+  superseding.** Previously it marked whatever `capture_id` it was handed as
+  superseded — so re-fetching an already-superseded version overwrote that
+  version's `superseded_by` pointer and orphaned the in-between versions,
+  leaving several un-superseded "current" captures for a single URL. It now
+  looks up the live un-superseded capture for the URL (`find_capture_by_url_hash`)
+  and supersedes that, so refetch always extends a single linear chain
+  regardless of which version it was invoked from (append-only supersession).
+
+### Changed — substrate
+
+- **Recall returns the current head of each supersession chain only** — one hit
+  per URL, not every refetched version. Superseded captures are pre-filtered out
+  of the ranking candidates (`storage_captures.superseded_capture_ids`) so they
+  don't consume top-k slots. Older versions remain on disk + in the DB
+  (retrievable by id); they're just out of the default recall surface.
+- **Superseded notes are self-describing in the vault.** When a refetch
+  supersedes a capture, the old note is stamped with a `superseded_by`
+  frontmatter field + an `[!info] Superseded` banner (reusing the visual-
+  indicator callout layer), so an older version reads as old when opened in
+  Obsidian. The current head note carries neither — only the DB pointer existed
+  before, which was invisible when reading the `.md`.
+
+## [0.1.5] — 2026-06-10
+
+Wayback archiving is now opt-in and quiet. A real-world end-to-end smoke of the
+non-Reddit extractors found the anonymous Wayback Machine submission — which ran
+on every capture — was both unreliable and slow.
+
+### Changed — substrate
+
+- **Wayback URL-archiving is now OPT-IN (default off).** archive.org's
+  anonymous Save-Page-Now is rate-limited by design (fast HTTP 429s) and
+  frequently hangs on the target-crawl for minutes, so unauthenticated
+  archiving cannot be relied on. It is no longer presented as a default capture
+  feature (don't-over-promise). Enable best-effort anonymous archiving with
+  `[archive] wayback_enabled = true` in `config.toml`. Reliable archiving needs
+  an authenticated archive.org account; a bring-your-own-credentials (SPN2 /
+  S3-key) reliable tier is planned (see roadmap).
+- **Wayback failures are now quiet.** A failed best-effort submission records
+  `archive_urls: {wayback: null}` in frontmatter but no
+  longer writes a `[!warning] Wayback archive failed` callout into the capture's
+  vault body — a tombstone over-stated an optional, known-unreliable witness.
+- **Wayback inline timeout lowered 30s → 15s.** When enabled, a hung anonymous
+  submission could block the user-facing capture for the full 30s (a live probe
+  measured a successful submission at ~9s and routine multi-minute hangs). 15s
+  keeps genuine successes while halving the worst-case wait.
+- **`/health` is honest about Wayback.** The wayback row is absent by default
+  (opt-in), and its docstring now states it is a *reachability-only* probe —
+  a green row means archive.org is reachable, **not** that archiving will
+  succeed (anonymous SPN is rate-limited/best-effort).
+
+## [0.1.4] — 2026-06-09
+
+Reddit HTML-channel fidelity: the two gaps v0.1.3 documented are now closed,
+both credential-free.
+
+### Changed — substrate
+
+- **Deep comment trees expand credential-free.** The old.reddit HTML channel
+  previously stopped at "load more comments" links and marked the thread
+  `comments_truncated`. It now follows them via the same `/api/morechildren`
+  endpoint the page's own JS uses (no credentials needed), re-nesting the flat
+  response under each comment's true parent — old.reddit flattens deep
+  continuations to the bottom of the page, so structural position can't be
+  trusted; the real parent comes back in the API response. Expansion is bounded
+  by a paced request budget (`max_more_requests`, default 24) to stay under
+  Reddit's rate-sensitive WAF; when the budget is spent (or a fetch fails, or a
+  link isn't parseable) the residual `more` markers keep `comments_truncated`
+  honestly `True`. Best-effort augmentation: any expansion failure leaves the
+  base capture intact.
+- **Galleries resolve to distinct full-resolution images.** Gallery media is
+  now read from old.reddit's authoritative `.gallery-tile[data-media-id]` list
+  (ordered, deduplicated) and mapped to full-res `i.redd.it/<id>.<ext>` URLs.
+  Previously the post tile's `preview.redd.it` thumbnails were scraped and
+  keyed by full URL, so the same image at several widths produced duplicate
+  gallery items pointing at thumbnails. Falls back to a preview-URL scrape
+  (deduped by media id) when the structured tiles are absent.
+
+Both channels still treat OAuth (when a Reddit app is configured) as the
+fidelity/rate upgrade: 60 req/min headroom for very large threads beyond the
+HTML budget, plus gallery dimensions/captions the HTML tiles don't expose.
+
+## [0.1.3] — 2026-06-09
+
+Credential-free Reddit capture. Reddit previously required every user to
+register their own OAuth app (`client_id`/`client_secret`) before any Reddit
+URL could be captured — a real adoption friction and the only source that
+reported `degraded` on a fresh install. Reddit's anonymous `.json` API is
+WAF-blocked, but the **old.reddit.com HTML** site still serves live threads +
+full comment trees to a browser-shaped request. v0.1.3 adds that as the
+**default** channel, so Reddit works out of the box; OAuth becomes an optional
+higher-fidelity / higher-rate-limit upgrade.
+
+### Changed — substrate
+
+- **`RedditExtractor` is now credential-free by default (shape A).** `extract()`
+  runs a fallback chain: when a Reddit app is configured, the OAuth-JSON channel
+  is tried first (cleaner comment pagination + 60 req/min headroom) and the
+  old.reddit-HTML channel backstops it; with no app, capture still works via
+  HTML. Both channels emit the same two-Listing shape, so all typed assembly
+  (`post_type` / media / `removed_status` / `is_op` / crosspost / comment
+  recursion) is shared verbatim. A bad/expired OAuth app now degrades gracefully
+  to HTML instead of failing the capture.
+- **`/health` for Reddit is green credential-free.** A missing Reddit app is no
+  longer `degraded` — only an unreachable old.reddit is. `fallback_count`
+  reflects the wired channels (2 with creds, 1 without).
+
+### Added — substrate
+
+- **Old.reddit HTML channel** — forces the `old.reddit.com` host + `?limit=500`
+  for the deep comment tree, sends the `over18` cookie to clear the NSFW
+  interstitial, and paces + retries on the rate-sensitive WAF's transient 403s.
+  Raw HTML is preserved to the Source-tier (`.html.gz`). Parses
+  post fields, self-text (rendered → markdown), media URLs, the recursive
+  comment tree, `removed`/`deleted` status, and truncation. *Known fidelity
+  gaps vs the OAuth channel: galleries are best-effort; very deep trees still
+  truncate (`comments_truncated`). old.reddit is self-host-only by design — a
+  datacenter IP would hit the WAF and `robots.txt`.*
+- **`RedditPayload.extractor_source`** gains `"reddit-html"` so the typed
+  payload self-describes which channel produced it.
+
+### CLI
+
+- **`khiipd capture` / `recall` / `refetch` no longer leak `httpx` tracebacks**
+  on a slow/timed-out request — they print a clean message (capture/refetch
+  note the daemon may still be finishing server-side) and exit non-zero. Default
+  `capture` / `refetch` timeouts raised to 120s so slow sources (YouTube,
+  Wikipedia) don't trip the old 30s/60s limits on the happy path.
+
+## [0.1.2] — 2026-06-08
+
+Dedicated Wikipedia extractor. Wikipedia URLs were previously captured by the
+generic `WebExtractor` (emitting a `WebPayload`); the `WikiPayload` type +
+renderer + embed-composition shipped at v0.1 but no extractor produced one.
+`WikiExtractor` completes that scaffolding by talking to the MediaWiki API
+directly — the one generic-web source with a first-class, versioned API.
+
+### Added — substrate
+
+- **`khiip.extractors.wiki.WikiExtractor`** — `*.wikipedia.org` `/wiki/<Title>`
+  article URLs (language subdomains + `.m.` mobile variants). Registered before
+  `WebExtractor` in the default registry so Wikipedia URLs are claimed here
+  rather than by the http(s) catch-all. Emits a typed `WikiPayload`
+  (`source="wiki"`).
+- **Two-source fallback chain** (the resilience pattern shared across all
+  extractors):
+  - **MediaWiki action API** (primary) — `action=query&prop=extracts|
+    pageimages|info&explaintext=1&exsectionformat=raw` for a clean plain-text
+    body parsed into structured `sections` (no fragile HTML scraping for the
+    load-bearing fields) + page image + canonical URL; `action=parse&prop=text`
+    rendered HTML parsed **best-effort** for `references` + `infobox` (the two
+    fields with no clean JSON surface — a parse miss degrades only those).
+  - **REST v1 summary** (fallback) — `/api/rest_v1/page/summary/{title}` for a
+    thinner lead-only payload via an orthogonal endpoint family.
+- **Source-tier raw-bytes preservation** — the structured query
+  JSON is gzipped + retained for future re-derivation.
+- **`health_check()`** probing the REST summary endpoint for the en "Wikipedia"
+  article (5s short-timeout), surfaced on `/health` like the other extractors.
+- **CC BY-SA 4.0 attribution** populated on every Wikipedia capture
+  (`contributors_attribution`), per Wikipedia's reuse license.
+
+### Notes
+
+- Foundation-vs-on-top preserved: new module + registration only; the v0.1.1
+  substrate (`WikiPayload`, `WikiMarkdownRenderer`, `_compose_wiki`,
+  `SourceName`) was already in place and is unchanged. +49 tests.
+
+## [0.1.1] — 2026-05-29
+
+MCP server. Six tools (`capture_url`, `recall`, `list_captures`,
+`get_capture`, `refetch_capture`, `daemon_status`) over stdio for Claude
+Desktop / Cursor / any MCP-aware client. Thin HTTP proxy architecture:
+the MCP server connects to a running Khiip
+daemon at `127.0.0.1:8478`, reads the Bearer auth token from
+`~/.config/khiip/auth.toml`, and translates MCP tool calls to REST.
+Daemon must be running before tool calls land.
+
+### Added — substrate
+
+- **`khiip.mcp` subpackage** — `DaemonClient` (httpx-based REST proxy),
+  `build_server` (FastMCP-based server factory with six tools registered),
+  `__main__` (entry point for the `khiip-mcp-server` console script).
+- **`khiip-mcp-server` console script** wired in `pyproject.toml`. Used by
+  Claude Desktop / Cursor MCP configs to launch the stdio server.
+- **`mcp>=1.27,<2` dependency** added to `dependencies` (bundled with the
+  daemon — no optional `[mcp]` extra). Aligns with the launch-essay +
+  Show HN claim that v0.1 ships REST + MCP together.
+- **Six MCP tools** mapping 1:1 to existing REST endpoints:
+  - `capture_url(url, force_new=False)` → `POST /api/v1/captures`
+  - `recall(query, limit=10)` → `GET /api/v1/recall`
+  - `list_captures(source=None, limit=50, offset=0)` → `GET /api/v1/captures`
+  - `get_capture(capture_id, format="json")` → `GET /api/v1/captures/{id}`
+  - `refetch_capture(capture_id, dimension="extraction")` → `POST /api/v1/captures/{id}/refetch`
+  - `daemon_status()` → `GET /health` + `GET /api/v1/meta` combined
+- **Structured error responses** — daemon-unreachable + HTTP-error
+  translation into LLM-readable `{"error": ..., "detail": ...}` dicts so
+  agents get actionable failure messages instead of opaque MCP errors.
+- **`KHIIP_DAEMON_URL` env var override** for pointing the MCP server at a
+  non-default daemon (useful for sandboxes / dev iterations).
+
+### Added — tests
+
+- `tests/test_mcp.py` (21 tests; +21 to v0.1.0's 747 baseline = 768 green
+  at v0.1.1). Coverage:
+  - `DaemonClient` unit tests against `httpx.MockTransport` (auth headers,
+    query-param threading, error-translation, env-var override)
+  - FastMCP tool tests with an injected fake client (argument threading,
+    return-shape envelopes, structured-error translation)
+  - Tool-surface lock: asserts exactly 6 tools registered at v0.1.1
+- End-to-end smoke verified by launching `khiip-mcp-server` via the
+  official `mcp` Python SDK client; `list_tools` returns all six.
+
+### Architectural notes
+
+- **REST stays canonical at v0.1.1.** MCP is a transport translator over
+  the REST surface. Service-tier refactor (extracting orchestration into
+  `khiip.services.*` functions both REST and MCP would call directly) is
+  deferred to a forcing function — a hosted/productized surface or
+  a third transport. The migration is low-cost: tool signatures, Claude
+  Desktop config, and behavior stay stable across the eventual refactor.
+- **Foundation-vs-on-top discipline:** all v0.1.1 changes live in the new
+  `khiip.mcp` subpackage. No existing module (extractors, renderers,
+  daemon routes, storage, embeddings, wayback) was modified. The v0.1.0
+  test suite continues to pass unchanged (747/747).
+
+### Stale-context cleanup
+
+- **README** — TRUST.md / "8 immutable promises" / "future MCP server"
+  references stripped; License section reframed from "Promise 1" to
+  LICENSE-grounded framing; status checklist updated for v0.1.1.
+
+## [0.1.0] — 2026-05-25
+
+First substantive substrate release. Six v0.1 sources (X, Reddit, Web,
+Wikipedia, YouTube, PDF) emit typed Pydantic payloads end-to-end; full
+Renderer Protocol+Registry tier; visual indicator system; Source-tier
+raw-bytes preservation; typed-payload recall composition; vault↔SQLite
+validator; granular refetch CLI; opt-in video preservation.
+
+### Added — substrate
+
+- **Typed-payload pipeline** — all 6 v0.1 sources emit
+  Pydantic-typed payloads at the extractor boundary: `TweetPayload`,
+  `RedditPayload`, `WikiPayload`, `WebPayload`, `PDFPayload`,
+  `YouTubePayload`, plus cross-platform primitives (`EngagementCounts`,
+  `UrlEntity`, `Media`, `CommentNode`) and X-specific `XArticle` /
+  `XArticleBlock`. Discriminated union over `kind`.
+- **Renderer Protocol + Registry** — 6 per-source
+  `MarkdownRenderer` impls + `LegacyMarkdownPassthroughRenderer` +
+  `JSONRenderer` + `VaultFrontmatterRenderer` dispatch first-supporting-
+  wins through `MarkdownRendererRegistry`.
+- **Source-tier raw-bytes preservation** (URI-shaped) — original
+  upstream bytes gzipped + written under
+  user-configurable `data_root` (default `~/.local/share/khiip/sources/
+  <source>/<capture_id>.<ext>.gz`); URI-shaped `source_artifact_path`
+  generalizes to cloud (`s3://`) / Notion (`notion://`) at v0.5+.
+- **MediaFetcherRegistry** — capability-based dispatch
+  across three fetchers: `HttpxFetcher` (photo CDN), `YtDlpFetcher`
+  (video via opt-in toggle; HLS/DASH/`v.redd.it`/YouTube),
+  `GalleryDlFetcher` (wide-coverage fallback).
+- **Wayback Machine archive submission** (`archive_urls`) —
+  best-effort URL archival on every capture; opt-out via
+  `[archive] wayback_enabled = false`.
+- **Visual indicator system** (three-layer model): layer 1
+  frontmatter-canonical, layer 2 renderer-composed Obsidian
+  callouts in vault body. Layer 3 plugin badges
+  deferred to v0.5+.
+- **Embed-text composition** — per-source typed-payload
+  composition rules for the embedder; URL-fallback for failed-permanent
+  tombstones (capture-what-we-can).
+- **Reddit OAuth2 channel** —
+  user registers their own Reddit app at https://www.reddit.com/prefs/apps;
+  client_id/secret via config.toml or env vars. Khiip-corporate never holds
+  upstream credentials.
+- **YouTube fallback chain** —
+  yt-dlp → youtube-transcript-api + oEmbed → YouTube Data API v3 (operator
+  opt-in via API key); per-branch payload-shape fidelity.
+- **Path-sandbox + defense-in-depth** at security boundaries: vault
+  `destination_path` validator, Source-tier
+  triple-guarded path validation (source/capture_id/ext rejection of
+  `..`/separators/NUL), media-fetcher subprocess argv-list invocation.
+
+### Added — surfaces
+
+- **`khiipd serve`** — FastAPI daemon (default `127.0.0.1:8478`).
+- **`khiipd capture <url>`** — capture a URL via the running daemon.
+- **`khiipd recall <query>`** — semantic recall against the embedded
+  corpus.
+- **`khiipd validate`** *(new in 0.1.0)* — vault ↔ SQLite consistency
+  checker. Four invariants: vault ↔ SQLite
+  bidirectional reconciliation, structured_payload Pydantic round-trip,
+  P-δ status family consistency, media local_path sandbox + existence.
+  Exit code 0 clean / 1 violations / 2 missing inputs. `--json` for
+  machine-readable output.
+- **`khiipd refetch <id> [--media|--wayback]`** *(new in 0.1.0)* —
+  granular refetch. Three dimensions
+  independently re-attemptable: extraction (default; re-run extractor;
+  new capture; old gets `superseded_by`), media (re-walk
+  MediaFetcherRegistry in place), wayback (re-submit + update
+  `archive_urls`).
+- **`khiipd auth show|rotate`** — manage the daemon API key.
+- **REST `?format=` selector** on `GET /api/v1/captures/{id}` —
+  `capture` (default; full JSON envelope) / `json` (payload only via
+  JSONRenderer) / `markdown` (vault file) / `legacy-markdown` (body
+  only).
+- **REST `POST /api/v1/captures/{id}/refetch?dimension=...`** *(new in
+  0.1.0)* — the daemon-side endpoint the CLI wraps.
+- **REST `?force_new=true`** on `POST /api/v1/captures` — bypass dedup;
+  used internally by refetch extraction dimension.
+
+### Added — configuration
+
+- `[storage] data_root` — user-configurable Source-tier root (default
+  XDG `~/.local/share/khiip/`). Points anywhere on a filesystem:
+  iCloud Drive / Dropbox / external SSD / network mount for
+  cross-machine sync.
+- `[media] download_videos` *(new in 0.1.0)* — opt-in default-off toggle
+  for video preservation. When `true`,
+  `YtDlpFetcher` registers in the default chain + `GalleryDlFetcher`
+  claims `media.type == "video"` (non-streaming-manifest URLs).
+- `[media] fetch_enabled` / `[media] fetcher_disabled` — per-source +
+  per-fetcher toggles.
+- `[archive] wayback_enabled` — Wayback opt-out (default true).
+- `[extractors.reddit] client_id` / `client_secret` (or
+  `KHIIP_REDDIT_CLIENT_ID` / `KHIIP_REDDIT_CLIENT_SECRET` env vars).
+- `[extractors.youtube] api_key` (or `KHIIP_YOUTUBE_API_KEY` env var).
+
+### Changed
+
+- **Reddit channel** revised from anonymous `.json` to OAuth2 against
+  `oauth.reddit.com`.
+- **Vault frontmatter is canonical** for the Payload tier; SQLite
+  `captures` + `embeddings` tables become derived caches rebuildable
+  from vault.
+- **`content_sha256`** rebases to hash the embed-text composition
+  so cosmetic rendering changes don't trigger re-embedding.
+
+### Documented
+
+- Foundational architecture decisions recorded: the graph schema, the
+  extractor Protocol, append-only edges, the Source/Payload/Render
+  three-tier model, and the P-δ failure-handling principle.
+- Substrate-vs-surfaces architectural axis: Khiip = one canonical substrate +
+  N surfaces (Obsidian plugin / REST / MCP / Chrome ext / mobile /
+  Telegram bot / LLM-agent SDK consumers all speak the unified contract).
+
+### Notes
+
+- This release lays the substrate baseline. Plugin UI rendering of the
+  new typed-payload + visual indicator + refetch surfaces lands at v0.5+
+  premium UX scope.
+- Pre-public-launch posture statement for SD-4 video-binary capability is
+  separately tracked (1-2 hr authoring; not blocking this tag).
+
+[0.2.0]: https://github.com/KhiipAI/khiip/releases/tag/v0.2.0

khiip-0.2.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,31 @@
+# Code of Conduct
+
+This project adopts the **[Contributor Covenant 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/)** as our Code of Conduct.
+
+The full text is available at the link above. In short: we expect all contributors and community members to engage respectfully, constructively, and with good faith — and to extend the same to others regardless of background, experience level, or identity.
+
+## Enforcement
+
+Instances of unacceptable behavior may be reported to the project maintainers at:
+
+**Email:** [hello@khiip.com](mailto:hello@khiip.com)
+
+All reports will be reviewed and investigated promptly and fairly. The project team is obligated to respect the privacy and security of the reporter of any incident.
+
+## Scope
+
+This Code of Conduct applies within all community spaces — including but not limited to:
+
+- The Khiip GitHub repository (issues, PRs, discussions, code comments)
+- Official Khiip communication channels (email, Discord when launched, etc.)
+- When an individual is officially representing the community in public spaces
+
+## Attribution
+
+This Code of Conduct adopts the Contributor Covenant, version 2.1, available at https://www.contributor-covenant.org/version/2/1/code_of_conduct/
+
+For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq
+
+---
+
+*Khiip Code of Conduct v1.0; last updated 2026-05-17. Adopts Contributor Covenant 2.1 by reference.*