ugraph-kit 0.1.0__py3-none-any.whl

ugraph/__init__.py

@@ -0,0 +1,28 @@
+"""
+ugraph-kit — build an agent-navigable knowledge base from any input you trust.
+
+Clipboard, paste, pipe, file, or URL. Plain markdown, YAML frontmatter, relative links.
+No database, no embeddings. An agent reads an index, follows links to the few pages it
+needs, and cites a chunk or timestamp in an immutable raw source.
+
+The format is the Open Knowledge Format, originated by Cole Medin
+(github.com/coleam00/cole-medin-knowledge-base). This package is an independent
+implementation of it as a reusable tool; divergences are marked OKF-v in SCHEMA.md.
+
+Library entry points:
+
+    from ugraph import config, indexes, ingest, lint, status, verify
+    from ugraph.sources import youtube
+
+    cfg = config.load(kb="~/vault/knowledge")
+    ingest.capture_text(cfg, "a claim you care about")   # any text
+    ingest.ingest_path(cfg, "./note.md")                 # any file
+    youtube.ingest(cfg, "https://youtube.com/@example", limit=25)
+    indexes.write_all(cfg)
+    findings, pages = lint.lint(cfg)
+    issues = verify.verify(cfg)
+"""
+
+__version__ = "0.1.0"
+
+__all__ = ["__version__"]

ugraph/_bundled/skills/channel-to-kb/SKILL.md

@@ -0,0 +1,194 @@
+---
+name: channel-to-kb
+description: Extract concepts and entities from ingested YouTube transcripts into an OKF knowledge base. Use when transcripts exist in raw/ with source pages marked summary_status pending, or when the user asks to process/extract/summarize ingested talks, grow the knowledge base from a channel, or run the extraction pass.
+---
+
+# Channel → Knowledge Base (extraction pass)
+
+Stage 2 of the KB pipeline. Stage 1 (`ugraph ingest`) already
+put timestamped transcripts in `raw/` and stub pages in `sources/`. This skill turns
+them into **canonical, cross-linked concept and entity pages**.
+
+Read `SCHEMA.md` at the KB root before writing anything. It is the contract;
+`ugraph lint` enforces it.
+
+## The one rule that matters
+
+**Synthesize across sources; never create one page per video.**
+
+A concept page is not a talk summary. It is the canonical statement of an idea,
+assembled from every source that taught it. Ten talks mentioning context engineering
+produce **one** `concepts/context-engineering.md` citing ten sources — not ten pages.
+
+If you find yourself creating `concepts/mcp-apps-talk.md`, stop. That is a source, and
+it already exists in `sources/`.
+
+## Two modes
+
+**Single-pass** (below) — read transcripts and write pages in one go. Correct for
+batches of ~10 or fewer.
+
+**Two-phase** — for a channel too large to hold in one context (100+ talks). The reason
+it exists: canonicalization needs a global view, but 750k tokens of transcript won't fit
+alongside page writing. Splitting lets the expensive reading go parallel while the
+decision that needs global knowledge stays serial.
+
+```
+Phase A  parallel, one agent per transcript   → <candidates>/<slug>.json
+         spec: references/candidate-extraction.md
+         emits candidates ONLY — never pages
+Phase B  serial, one context                  → cluster candidates, decide
+         create / merge / embed against the existing concepts/index.md
+Phase C  parallel, ONE AGENT PER CONCEPT      → write pages
+         parallelising by concept, not transcript, makes write conflicts impossible
+Phase D  serial                               → reciprocity, ugraph index, ugraph lint
+```
+
+**Never parallelise Phase C by transcript.** Twenty agents reading twenty talks will each
+independently create `context-engineering.md`, and the merge — the entire value of the
+KB — is lost.
+
+Check progress with `ugraph status --clusters`, or `ugraph ledger --stuck 0` for the
+per-item work queue.
+
+## Workflow (single-pass, and Phase B/C of two-phase)
+
+### 1. Pick the batch
+
+```bash
+ugraph ledger --pending --limit 10
+```
+
+Default batch size is **10 transcripts**. Larger batches degrade canonicalization —
+you stop noticing that talk 14 and talk 3 are describing the same idea.
+
+### 2. Load what already exists — before reading any transcript
+
+Read `concepts/index.md` and `entities/index.md` at the KB root, or:
+
+```bash
+ugraph status --thin        # concepts with one source — the likeliest merge targets
+```
+
+This is the dedup baseline. You cannot canonicalize against pages you haven't seen,
+and the most common failure of this pass is creating a near-duplicate of a concept that
+already exists under a slightly different name.
+
+### 3. Read the transcripts
+
+Read each `raw/` file in the batch. For each, note:
+
+- **Claims worth keeping** — with their `[HH:MM:SS]` timestamps
+- **Named things** — tools, people, companies → candidate entities
+- **Ideas** — techniques, patterns, arguments → candidate concepts
+
+Ignore: conference logistics, speaker intros, demo narration, audience Q&A chatter.
+A 20-minute talk usually yields 1–3 real concepts. Often zero. **Zero is a valid
+result** — say so rather than manufacturing a page.
+
+### 4. Canonicalize
+
+For each candidate, decide:
+
+| Situation | Action |
+|---|---|
+| Page already exists | **Merge** — add the new source's angle + citation to the existing page. Update `sources:` and `updated:`. |
+| New, and appears in ≥2 sources *or* linked from ≥2 places | **Create** a new page |
+| New, appears once, not linked elsewhere | **Embed** it in a parent concept — do not create a page |
+
+That last row is the page-creation threshold from SCHEMA.md. Respect it. A KB of
+single-mention stubs is worse than a smaller dense one.
+
+### 5. Write the pages
+
+Follow SCHEMA.md exactly:
+
+- Frontmatter: `type`, `title`, one-sentence `description`, `domain` (closed vocabulary
+  in `taxonomy.json`), `status`, `sources`, `created`, `updated`
+- **Relative markdown links only.** No `[[wikilinks]]` anywhere in the OKF tree.
+- Typed relationship headings: `## Prerequisites`, `## Builds on`, `## Contrasts with`,
+  `## Implemented by`, `## Related`, `## Sources`
+- **Reciprocate every typed edge.** If A links to B under a typed heading, add the
+  matching link on B. The linter warns on one-way edges.
+- Cite claims: `([Talk title](../sources/ai-engineer/slug.md) @ 00:14:32)`
+
+### 6. Update the source pages you consumed
+
+For each source in the batch:
+
+- Replace the placeholder `description` with a real one-sentence thesis
+- Set `summary_status: done`
+- Replace the stub body with a short summary and a `## Concepts extracted` list
+  linking to the concept pages you wrote
+
+### 7. Record what you did
+
+The lifecycle ledger derives current state from the files, but it cannot know *when* a
+stage happened or *why* something was set aside. Phases A–C run in an agent rather than
+in Python, so those transitions are only in the ledger if you put them there:
+
+```bash
+ugraph ledger record <source-slug> extracted   --by "phase A"
+ugraph ledger record <source-slug> synthesized --by "phase C" --detail "fed 3 concepts"
+ugraph ledger record <source-slug> skipped     --by "phase A" --detail "vendor pitch, no transferable content"
+```
+
+`skipped` matters as much as the others. A talk with nothing in it is *finished*, not
+pending, and without that record it sits in the backlog forever being re-read.
+
+If you defer a talk to a different cluster, record `extracted` with a detail saying
+where it went. That is the difference between "in flight" and "forgotten".
+
+### 8. Validate
+
+```bash
+ugraph index
+ugraph lint
+ugraph verify
+```
+
+All three must pass before you report done — `lint` and `verify` with **0 errors**. Fix what it reports; do not
+hand back a failing KB. Orphan warnings on sources you just processed mean you didn't
+link them from a concept — go back to step 5.
+
+### 9. Report
+
+Tell the user, concretely:
+
+- Concepts **created** vs **merged into** (these are different, and merges are the
+  signal the KB is working)
+- Entities added
+- Transcripts that yielded nothing, and why
+- Current lint status
+
+Then stop. Let them review the diff before the next batch.
+
+## Quality bar
+
+The pages you write should read like the hand-built ones in `concepts/` — for example
+any well-formed page already in `concepts/`. Specifically:
+
+- A `>` blockquote opening line that states the idea sharply
+- Prose that explains *why it matters*, not a bulleted transcript restatement
+- Genuine cross-links that a reader would actually follow
+- Every non-obvious claim traceable to a source and timestamp
+
+**Do not pad.** If a concept only warrants four sentences, write four sentences.
+A short true page beats a long padded one.
+
+## What not to do
+
+- Don't invent claims the transcript doesn't support. Auto-captions garble names and
+  terms — if a term looks mangled, verify against the video or leave it out.
+- Don't create a concept page per talk.
+- Don't use `[[wikilinks]]` in the OKF tree.
+- Don't edit anything in `raw/`. It is immutable provenance.
+- Don't mark `summary_status: done` on a source you didn't actually read.
+- Don't skip the lint step.
+
+## Related
+
+- Stage 1 ingestion — `ugraph ingest`
+- Schema contract — `SCHEMA.md` at the KB root
+- Conformance gate — `ugraph lint` · quote verification — `ugraph verify`
+- Index generation — `ugraph index` · lifecycle — `ugraph ledger`

ugraph/_bundled/skills/channel-to-kb/references/candidate-extraction-text.md

@@ -0,0 +1,39 @@
+# Phase A candidate extraction — text documents (chunk-anchored)
+
+You are extracting candidate concepts from one pasted text document. The document is
+given to you as numbered chunks (`--- CHUNK <id> ---`). This is mechanical work:
+find the claims, copy the words exactly. Do not synthesize, do not editorialize, do
+not merge ideas across the document.
+
+## Rules
+
+1. `verbatim_quote` MUST be an exact substring of ONE chunk, copied
+   character-for-character (whitespace may differ). Never quote across a chunk
+   boundary. If you cannot quote it, it is not a concept — it is your paraphrase,
+   and it will be rejected by a substring test before anything is written.
+2. Do NOT invent anchors or timestamps. The pipeline derives the anchor (chunk id)
+   from your quote. You only supply name, claim, and quote.
+3. `claim` is one sentence: what the text asserts, in your words. The claim may
+   paraphrase; the quote may not.
+4. `name` is a short concept name (e.g. "reciprocal rank fusion").
+5. Prefer 3–8 strong concepts over 20 weak ones. Marketing copy, a changelog of
+   trivia, or content-free notes are `yield: "none"` with an empty list — that is a
+   correct answer.
+6. Tables count as text: quoting one full row is fine if it sits inside one chunk.
+
+## Output
+
+Return ONLY this JSON object, no prose, no code fence:
+
+```json
+{
+  "yield": "high | medium | low | none",
+  "concepts": [
+    {
+      "name": "concept name",
+      "claim": "one sentence: what is asserted",
+      "verbatim_quote": "exact substring from one chunk"
+    }
+  ]
+}
+```

ugraph/_bundled/skills/channel-to-kb/references/candidate-extraction.md

@@ -0,0 +1,115 @@
+# Phase A — candidate extraction (subagent spec)
+
+You read **one transcript** and emit **one JSON file**. You do not write knowledge base
+pages. You do not decide whether a concept deserves a page — that decision needs a view
+across the whole corpus, which you don't have.
+
+Your output is the raw material a later canonicalization pass clusters. Precision matters
+more than coverage: a wrong quote poisons a page that cites it.
+
+## Input / output
+
+- **Read:** `<kb>/raw/<channel>/<slug>.md`
+- **Write:** the KB's candidates directory — `ugraph status --json`
+  reports it, and it defaults to `.ugraph/candidates/<slug>.json`
+
+Nothing else. Do not touch anything under the knowledge base itself.
+
+## Schema
+
+```json
+{
+  "slug": "ai-engineer/<slug>",
+  "title": "exact title from the transcript frontmatter",
+  "yield": "high | low | none",
+  "thesis": "One sentence: the argument this talk makes. Not a topic label.",
+  "cluster_hint": "harness | memory-context | rl-posttraining | evals | security | fde | multi-agent | long-horizon | meta-role | ux-product | local-edge | vertical | mcp | other",
+  "concepts": [
+    {
+      "name": "short lowercase noun phrase",
+      "claim": "One sentence stating what the talk asserts about this.",
+      "verbatim_quote": "Exact words copied from the transcript.",
+      "timestamp": "00:14:32",
+      "domain": "agentic_systems | ai_engineering | rag | local_llms | machine_learning | mathematics | system_design | product"
+    }
+  ],
+  "entities": [
+    {"name": "TauBench", "subtype": "tool", "note": "one line on what it is / why it matters here"}
+  ],
+  "notes": "Optional. Caveats, garbled sections, anything the canonicalizer should know."
+}
+```
+
+## Rules
+
+**Quotes must be verbatim.** Copy exactly from the transcript — do not clean up grammar,
+do not paraphrase, do not stitch two sentences together. The quote is what a later page
+will cite, and someone will click the timestamp to check it. Auto-captions are messy;
+that's fine, copy the mess.
+
+**Timestamps must be real.** Use the `[HH:MM:SS]` marker of the paragraph the quote came
+from. Never estimate.
+
+**Watch for garbled names.** These are machine captions. Speaker names, company names, and
+product names are frequently mangled — real examples: "Aruba"/"Rue Ba" for Uber,
+"SER"/"Sonder" for SonderMind, "Sweet Bench" for SWE-Bench. If a term looks corrupted and
+you cannot confidently recover it, **omit it or flag it in `notes`. Never guess a name.**
+
+**`yield: none` is a correct answer.** Track intros, sponsor pitches, workshop logistics,
+and demo narration frequently contain no transferable idea. Return `"yield": "none"` with
+an empty `concepts` array and say why in `notes`. Do not manufacture concepts to look
+productive. A typical 20-minute conference talk yields **1–3** real concepts; many yield
+zero.
+
+**Name concepts generically, not by talk.** `"context compaction"`, not
+`"Notion's approach to context compaction"`. The canonicalizer merges by name, so
+talk-specific names defeat the whole point. If two talks describe the same idea, they
+should produce the same or similar `name`.
+
+**One entry per distinct idea.** Don't split one idea into four near-identical entries to
+pad the list, and don't merge two genuinely different ideas into one.
+
+## What counts as a concept
+
+Include a technique, pattern, architectural decision, failure mode, or argument that
+would still be useful to someone who never watches this talk.
+
+Exclude: product announcements, company background, conference logistics, "come to our
+booth", personal anecdotes without a transferable point, and restatements of things every
+practitioner already knows.
+
+**Read the Q&A. Weight it up, not down.** This is counterintuitive and it was learned the
+hard way: on a 94-minute workshop, ~60% of the runtime was setup and live-coding narration
+that yielded nothing — but the audience Q&A was the single richest vein in the file, and
+four of eleven concepts came from it. Prepared talks are rehearsed and often pitch-shaped.
+Q&A is unscripted practitioners asking about the thing that actually bit them
+("when do you copy data into a graph versus leave it in place?"). A speaker rarely
+volunteers that. Never skip the last third of a transcript because the prepared portion
+has ended.
+
+**Judge the source, not just the content.** If a talk is a vendor pitch with a
+predetermined conclusion, or rests on a demo rather than a benchmark, say so in `notes`.
+Extract the ideas anyway — but the canonicalizer needs to know whether a claim is backed
+by production numbers at scale or by one person's home lab, because that decides whether
+the resulting page carries `confidence: low`.
+
+**The speaker can be wrong.** Verbatim quoting protects against caption noise; it does not
+protect against factual error. One talk glossed OWL as "web object language" (it is Web
+Ontology Language). Record the claim accurately, flag the error in `notes`, and never
+propagate it as fact.
+
+## Yield levels
+
+| Level | Meaning |
+|---|---|
+| `high` | 2+ concepts a practitioner could act on |
+| `low` | 1 concept, or ideas that are real but thin |
+| `none` | nothing transferable |
+
+## Before you finish
+
+- Every `verbatim_quote` appears character-for-character in the transcript
+- Every `timestamp` matches a real `[HH:MM:SS]` marker in that file
+- Every `domain` is from the closed list above
+- The JSON parses
+- You wrote exactly one file, to the candidates directory

ugraph/_bundled/templates/SCHEMA.md

@@ -0,0 +1,177 @@
+---
+type: overview
+title: "Knowledge Base Schema (OKF-v)"
+description: "Page types, frontmatter contracts, relationships, and linking rules for this knowledge base."
+---
+
+# Knowledge Base Schema — OKF-v
+
+This knowledge base uses plain Markdown, YAML frontmatter, generated indexes,
+and relative links. Raw source material remains inspectable; derived pages link
+back to the sources that support them.
+
+## Directory layout
+
+```text
+knowledge/
+├── index.md
+├── SCHEMA.md
+├── taxonomy.json
+├── concepts/
+├── entities/
+│   ├── tools/
+│   ├── people/
+│   └── organizations/
+├── sources/
+├── raw/
+└── _mocs/
+```
+
+`concepts/` is flat. Subject grouping is represented by the `domain` field,
+which keeps file paths stable when classification changes.
+
+## Page contracts
+
+All dates use `YYYY-MM-DD`.
+
+### Concept
+
+Location: `concepts/<slug>.md`
+
+```yaml
+---
+type: concept
+title: "Generation-verification loop"
+description: "A generator proposes outputs and an independent gate rejects unsupported ones."
+domain: agentic_systems
+status: growing
+tags: [verification]
+sources: [example/source-slug]
+created: 2026-01-01
+updated: 2026-01-01
+---
+```
+
+Required fields: `type`, `title`, `description`, `domain`, `status`, `created`,
+and `updated`. Status is one of `seed`, `growing`, or `evergreen`.
+
+### Entity
+
+Location: `entities/{tools,people,organizations}/<slug>.md`
+
+```yaml
+---
+type: entity
+subtype: person
+title: "Example Person"
+description: "A short, factual description."
+resource: https://example.com
+handles: ["@example"]
+created: 2026-01-01
+updated: 2026-01-01
+---
+```
+
+Required fields: `type`, `subtype`, `title`, `description`, `created`, and
+`updated`. Subtype is one of `tool`, `person`, or `organization`.
+
+### Source
+
+Location: `sources/<publisher>/<slug>.md`
+
+```yaml
+---
+type: source
+source_type: article
+title: "Example source"
+description: "The source's central claim in one sentence."
+slug: example/source
+url: https://example.com/source
+created: 2026-01-01
+updated: 2026-01-01
+---
+```
+
+Required fields: `type`, `source_type`, `title`, `description`, `slug`,
+`created`, and `updated`. Video sources additionally require `youtube_id`,
+`url`, `published`, `duration`, and `raw`.
+
+### Raw transcript
+
+Location: `raw/<publisher>/<slug>.md`
+
+```yaml
+---
+type: raw-transcript
+immutable: true
+slug: example/source
+url: https://example.com/source
+---
+```
+
+Never edit generated raw transcripts by hand. Timestamped transcript blocks use:
+
+```text
+[00:04:12] The exact source text appears here.
+```
+
+### MOC and overview
+
+Maps of content use `type: moc`. Entry points, redirects, and this schema use
+`type: overview`. Both require `type` and `title`.
+
+## Relationships
+
+A Markdown link under one of these headings asserts an edge:
+
+- `## Prerequisites`
+- `## Builds on`
+- `## Contrasts with`
+- `## Implemented by`
+- `## Related`
+- `## Sources`
+
+Typed edges should be linked in both directions where that relationship is
+meaningful. Provenance links from concepts to sources do not require a forward
+link from every source.
+
+## Linking rules
+
+- Use relative Markdown links inside the knowledge base.
+- Do not use `[[wikilinks]]` in the strict tree (`concepts`, `entities`,
+  `sources`, and `_mocs`).
+- Every link must resolve.
+- Filenames are stable identities; use kebab-case and do not rename casually.
+- Prefer one topic per page.
+
+## Citation rule
+
+Claims from timestamped material cite the source and exact time:
+
+```markdown
+The speaker describes a verification loop
+([Example talk](../sources/example/talk.md) @ 00:14:32).
+```
+
+Text extraction candidates carry a content-addressed chunk anchor. ugraph's
+verification gate rejects quotes that do not occur verbatim in the source.
+
+## Domains
+
+The closed vocabulary is defined in `taxonomy.json`. The default domains are:
+
+`agentic_systems`, `ai_engineering`, `rag`, `local_llms`,
+`machine_learning`, `mathematics`, `system_design`, and `product`.
+
+## Validation
+
+Run:
+
+```bash
+ugraph lint
+ugraph verify
+```
+
+Errors include malformed frontmatter, unknown closed-vocabulary values, broken
+links, invalid strict-tree wikilinks, and source pages whose raw target is
+missing. Warnings identify issues such as orphan pages and one-way typed edges.

ugraph/_bundled/templates/taxonomy.json

@@ -0,0 +1,44 @@
+{
+  "_comment": "Closed vocabulary used for validation and generated index grouping.",
+  "domains": {
+    "agentic_systems": "Agentic Systems",
+    "ai_engineering": "AI Engineering",
+    "rag": "Retrieval & RAG",
+    "local_llms": "Local LLMs",
+    "machine_learning": "Machine Learning",
+    "mathematics": "Mathematics",
+    "system_design": "System Design",
+    "product": "Product & Strategy"
+  },
+  "domain_order": [
+    "agentic_systems",
+    "ai_engineering",
+    "rag",
+    "local_llms",
+    "machine_learning",
+    "mathematics",
+    "system_design",
+    "product"
+  ],
+  "entity_subtypes": {
+    "tool": "Tools",
+    "person": "People",
+    "organization": "Organizations"
+  },
+  "entity_dirs": {
+    "tool": "tools",
+    "person": "people",
+    "organization": "organizations"
+  },
+  "source_types": {
+    "video": "Videos",
+    "talk": "Talks",
+    "paper": "Papers",
+    "article": "Articles",
+    "thread": "Threads",
+    "course": "Courses",
+    "book": "Books"
+  },
+  "moc_for_domain": {},
+  "page_types": {}
+}