@0xkahi/llw 0.0.0-stage → 0.0.0

package/skill-data/llw-okf/sk-references/spec-v01.md

@@ -0,0 +1,451 @@
+# Open Knowledge Format (OKF)
+
+**Version 0.1 — Draft**
+
+OKF is an open, human- and agent-friendly format for representing
+*knowledge* — the metadata, context, and curated insight that surrounds
+data and systems. It is designed to be authored by people, generated by
+agents, exchanged across organizations, and consumed by both.
+
+The format is intentionally minimal: a directory of markdown files with
+YAML frontmatter. There is no schema registry, no central authority, and
+no required tooling. If you can `cat` a file, you can read OKF; if you
+can `git clone` a repo, you can ship it.
+
+---
+
+## 1. Motivation
+
+The space of knowledge representation for AI agents is evolving quickly,
+and many incompatible conventions are emerging. OKF takes the position
+that knowledge is best represented in commonly accessible, established
+formats that are:
+
+- **Readable** by humans without tooling.
+- **Parseable** by agents without bespoke SDKs.
+- **Diffable** in version control.
+- **Portable** across tools, organizations, and time.
+
+The format is minimally opinionated. It standardizes only the small set
+of structural conventions needed to make a knowledge corpus
+*self-describing* — anything beyond that is left to the producer.
+
+### Goals
+
+1. Define a universal format that **enrichment agents** can write into.
+2. Inform how **consumption agents** should read and traverse it.
+3. Facilitate **exchange** of knowledge across systems and organizations.
+4. Standardize the small number of **required** fields that must be
+   present for content to be meaningfully consumed.
+
+### Non-goals
+
+- Defining a fixed taxonomy of concept types.
+- Prescribing storage, serving, or query infrastructure.
+- Replacing domain-specific schemas (Avro, Protobuf, OpenAPI, etc.) —
+  OKF *references* them; it does not subsume them.
+
+---
+
+## 2. Terminology
+
+- **Knowledge Bundle** — A self-contained, hierarchical collection of
+  knowledge documents. The unit of distribution.
+- **Concept** — A single unit of knowledge within a bundle. Represented
+  as one markdown document. May describe a tangible asset (a table, an
+  API), an abstract idea (a metric, a business process), or anything in
+  between.
+- **Concept ID** — The path of the concept's file within the bundle,
+  with the `.md` suffix removed. For example, `tables/users.md` has
+  concept ID `tables/users`.
+- **Frontmatter** — YAML metadata block delimited by `---` at the top of
+  a markdown file.
+- **Body** — Everything in the file after the frontmatter.
+- **Link** — A standard markdown link from one concept to another, used
+  to express relationships beyond the implicit parent/child hierarchy.
+- **Citation** — A link from a concept to an external source that
+  supports a claim in the body.
+
+---
+
+## 3. Bundle Structure
+
+A bundle is a directory tree of markdown files. The directory structure
+is independent of the domain — producers organize concepts however makes
+sense for the knowledge being captured.
+
+```
+path/to/bundle/
+├── index.md                      # Optional. Directory listing for progressive disclosure.
+├── log.md                        # Optional. Chronological history of updates.
+├── <concept>.md                  # A concept at the bundle root.
+└── <subdirectory>/               # Subdirectories organize concepts into groups.
+    ├── index.md
+    ├── <concept>.md
+    └── <subdirectory>/
+        └── …
+```
+
+A bundle MAY be distributed as:
+
+- A git repository (recommended — provides history, attribution, diffs).
+- A tarball or zip archive of the directory.
+- A subdirectory within a larger repository.
+
+### 3.1 Reserved filenames
+
+The following filenames have defined meaning at any level of the
+hierarchy and MUST NOT be used for concept documents:
+
+| Filename     | Purpose                                                |
+|--------------|--------------------------------------------------------|
+| `index.md`   | Directory listing. See §6.                             |
+| `log.md`     | Update history. See §7.                                |
+
+All other `.md` files are concept documents.
+
+Tags themselves remain a first-class concept — see the `tags`
+frontmatter field in §4.1. OKF does not specify a separate file format
+for aggregating documents by tag; producers that want a tag-browsing
+view can synthesize one at consumption time by scanning frontmatter.
+
+---
+
+## 4. Concept Documents
+
+Every concept is a UTF-8 markdown file. It has two parts:
+
+1. A **YAML frontmatter block**, delimited by `---` on its own line at
+   the start of the file and a closing `---` on its own line.
+2. A **markdown body**, containing free-form content.
+
+### 4.1 Frontmatter
+
+```yaml
+---
+type: <Type name>                  # REQUIRED
+title: <Optional display name>
+description: <Optional one-line summary>
+resource: <Optional canonical URI for the underlying asset>
+tags: [<tag>, <tag>, …]            # Optional
+timestamp: <ISO 8601 datetime>     # Optional last-modified time
+# … other producer-defined key/value pairs
+---
+```
+
+**Required:**
+
+- `type` — A short string identifying the kind of concept. Consumers
+  use this for routing, filtering, and presentation. Example values:
+  `BigQuery Table`, `BigQuery Dataset`, `API Endpoint`, `Metric`,
+  `Playbook`, `Reference`.
+
+  Type values are **not** registered centrally. Producers SHOULD pick
+  values that are descriptive and self-explanatory; consumers MUST
+  tolerate unknown types gracefully (typically by treating them as
+  generic concepts).
+
+**Recommended (in priority order):**
+
+- `title` — Human-readable display name. If omitted, consumers MAY
+  derive a title from the filename.
+- `description` — A single sentence summarizing the concept. Used by
+  `index.md` generators, search snippets, and previews.
+- `resource` — A URI that uniquely identifies the underlying asset the
+  concept describes. Absent for concepts that describe abstract ideas
+  rather than physical resources.
+- `tags` — A YAML list of short strings for cross-cutting categorization.
+- `timestamp` — ISO 8601 datetime of last meaningful change.
+
+**Extensions:** Producers MAY include any additional keys. Consumers
+SHOULD preserve unknown keys when round-tripping and SHOULD NOT reject
+documents with unrecognized fields.
+
+### 4.2 Body
+
+The body is standard markdown. Producers SHOULD favor structural
+markdown — headings, lists, tables, fenced code blocks — over freeform
+prose, since structure aids both human reading and agent retrieval.
+
+There are no required body sections. The following section headings have
+**conventional** meaning and SHOULD be used when applicable:
+
+| Heading        | Purpose                                                |
+|----------------|--------------------------------------------------------|
+| `# Schema`     | Structured description of an asset's columns/fields.   |
+| `# Examples`   | Concrete usage examples, often as fenced code blocks.  |
+| `# Citations`  | External sources backing claims in the body. See §8.   |
+
+### 4.3 Example: a concept bound to a resource
+
+```markdown
+---
+type: BigQuery Table
+title: Customer Orders
+description: One row per completed customer order across all channels.
+resource: https://console.cloud.google.com/bigquery?p=acme&d=sales&t=orders
+tags: [sales, orders, revenue]
+timestamp: 2026-05-28T14:30:00Z
+---
+
+# Schema
+
+| Column        | Type      | Description                              |
+|---------------|-----------|------------------------------------------|
+| `order_id`    | STRING    | Globally unique order identifier.        |
+| `customer_id` | STRING    | Foreign key into [customers](/tables/customers.md). |
+| `total_usd`   | NUMERIC   | Order total in US dollars.               |
+| `placed_at`   | TIMESTAMP | When the customer submitted the order.   |
+
+# Joins
+
+Joined with [customers](/tables/customers.md) on `customer_id`.
+
+# Citations
+
+[1] [BigQuery table schema](https://console.cloud.google.com/bigquery?p=acme&d=sales&t=orders)
+```
+
+### 4.4 Example: a concept not bound to a resource
+
+```markdown
+---
+type: Playbook
+title: Incident response — data freshness alert
+description: Steps to triage a freshness alert on the orders pipeline.
+tags: [oncall, incident]
+timestamp: 2026-04-12T09:00:00Z
+---
+
+# Trigger
+
+A freshness alert fires when `orders` lags more than 30 minutes behind
+its expected SLA. See the [orders table](/tables/orders.md).
+
+# Steps
+
+1. Check the [ingestion job dashboard](https://example.com/dash).
+2. …
+```
+
+---
+
+## 5. Cross-linking
+
+Concepts MAY link to other concepts using standard markdown links. Two
+forms are supported:
+
+### 5.1 Absolute (bundle-relative) links
+
+Begin with `/`, interpreted relative to the bundle root.
+
+```markdown
+See the [customers table](/tables/customers.md) for the join key.
+```
+
+This is the **recommended** form because it is stable when documents are
+moved within their subdirectory.
+
+### 5.2 Relative links
+
+Standard markdown relative paths.
+
+```markdown
+See the [neighboring concept](./other.md).
+```
+
+### 5.3 Link semantics
+
+A link from concept A to concept B asserts a *relationship*. The
+specific kind of relationship (parent/child, references, joins-with,
+depends-on, etc.) is conveyed by the surrounding prose, not by the link
+itself. Consumers that build a graph view typically treat all links as
+directed edges of an untyped relationship.
+
+Consumers MUST tolerate broken links — a link whose target does not
+exist in the bundle is not malformed; it may simply represent
+not-yet-written knowledge.
+
+---
+
+## 6. Index Files
+
+An `index.md` file MAY appear in any directory, including the bundle
+root. It enumerates the directory's contents to support **progressive
+disclosure** — letting a human or agent see what is available before
+opening individual documents.
+
+Index files contain no frontmatter. The body uses one or more sections,
+each grouping concepts under a heading:
+
+```markdown
+# Section / Group Heading
+
+* [Title 1](relative-url-1) - short description of item 1
+* [Title 2](relative-url-2) - short description of item 2
+
+# Another Section
+
+* [Subdirectory](subdir/) - short description of the subdirectory
+```
+
+Entries SHOULD include the description from the linked concept's
+frontmatter. Producers MAY generate `index.md` automatically; consumers
+MAY synthesize one on the fly when none is present.
+
+---
+
+## 7. Log Files (optional)
+
+A `log.md` file MAY appear at any level of the hierarchy to record the
+history of changes to that scope. The format is a flat list of
+date-grouped entries, newest first:
+
+```markdown
+# Directory Update Log
+
+## 2026-05-22
+* **Update**: Added new BigQuery table reference for [Customer Metrics](/tables/customer-metrics.md).
+* **Creation**: Established the [Dataplex Playbook](/playbooks/dataplex.md).
+
+## 2026-05-15
+* **Initialization**: Created foundational directory structure.
+* **Update**: Added progressive-disclosure guidelines to the root [index](/index.md).
+```
+
+Date headings MUST use ISO 8601 `YYYY-MM-DD` form. Log entries are
+prose; the leading bold word (`**Update**`, `**Creation**`,
+`**Deprecation**`, etc.) is a convention, not a requirement.
+
+---
+
+## 8. Citations
+
+When a concept's body makes claims sourced from external material,
+those sources SHOULD be listed under a `# Citations` heading at the
+bottom of the document, numbered:
+
+```markdown
+# Citations
+
+[1] [BigQuery public dataset announcement](https://cloud.google.com/blog/products/data-analytics/...)
+[2] [Internal data quality runbook](https://wiki.acme.internal/data/quality)
+```
+
+Citation links MAY be absolute URLs, bundle-relative paths, or paths
+into a `references/` subdirectory that mirrors external material as
+first-class OKF concepts.
+
+---
+
+## 9. Conformance
+
+A bundle is **conformant** with OKF v0.1 if:
+
+1. Every non-reserved `.md` file in the tree contains a parseable YAML
+   frontmatter block.
+2. Every frontmatter block contains a non-empty `type` field.
+3. Every reserved filename (`index.md`, `log.md`) follows the structure
+   described in §6 and §7 respectively when present.
+
+Consumers SHOULD treat all other constraints as soft guidance. In
+particular, consumers MUST NOT reject a bundle because of:
+
+- Missing optional frontmatter fields.
+- Unknown `type` values.
+- Unknown additional frontmatter keys.
+- Broken cross-links.
+- Missing `index.md` files.
+
+This permissive consumption model is intentional: OKF is meant to
+remain useful as bundles grow, get refactored, and are partially
+generated by agents.
+
+---
+
+## 10. Relationship to other formats
+
+OKF is intentionally close to several established patterns:
+
+- **LLM "wiki" repositories** that use markdown + frontmatter as
+  agent-readable knowledge bases.
+- **Personal knowledge tools** like Obsidian and Notion, which use
+  hierarchical markdown with cross-links.
+- **"Metadata as code"** approaches that store catalog metadata
+  alongside source code rather than in a separate registry.
+
+OKF differs primarily in being **specified** — pinning down the small
+set of rules needed for interoperability without dictating tooling.
+
+---
+
+## 11. Versioning
+
+This document specifies OKF version **0.1**. Future revisions will be
+versioned in the form `<major>.<minor>`:
+
+- A **minor** version bump introduces backward-compatible additions
+  (new optional fields, new conventional section headings).
+- A **major** version bump may make breaking changes (renaming required
+  fields, changing reserved filenames).
+
+Bundles MAY declare the OKF version they target by including
+`okf_version: "0.1"` in a bundle-root `index.md` frontmatter block (the
+only place frontmatter is permitted in an `index.md`). Consumers that
+do not understand the declared version SHOULD attempt best-effort
+consumption rather than refusing the bundle.
+
+---
+
+## Appendix A — Minimal example bundle
+
+```
+my_bundle/
+├── index.md
+├── datasets/
+│   ├── index.md
+│   └── sales.md
+└── tables/
+    ├── index.md
+    ├── orders.md
+    └── customers.md
+```
+
+`datasets/sales.md`:
+
+```markdown
+---
+type: BigQuery Dataset
+title: Sales
+description: All sales-related tables for the retail business.
+resource: https://console.cloud.google.com/bigquery?p=acme&d=sales
+tags: [sales]
+timestamp: 2026-05-28T00:00:00Z
+---
+
+The sales dataset contains transactional tables, including
+[orders](/tables/orders.md) and [customers](/tables/customers.md).
+```
+
+`tables/orders.md`:
+
+```markdown
+---
+type: BigQuery Table
+title: Orders
+description: One row per completed customer order.
+resource: https://console.cloud.google.com/bigquery?p=acme&d=sales&t=orders
+tags: [sales, orders]
+timestamp: 2026-05-28T00:00:00Z
+---
+
+# Schema
+
+| Column        | Type      | Description                  |
+|---------------|-----------|------------------------------|
+| `order_id`    | STRING    | Unique order identifier.     |
+| `customer_id` | STRING    | FK to [customers](/tables/customers.md). |
+| `total_usd`   | NUMERIC   | Order total in USD.          |
+
+Part of the [sales dataset](/datasets/sales.md).
+```

package/skill-data/llw-okf-sync/SKILL.md

@@ -0,0 +1,236 @@
+---
+name: llw-okf-sync
+description: >
+  Sync all Knowledge Bundles across the `llm-wiki` vault, updating, out of date bundle.md files,
+  with the latest content based on their concepts and child bundles. Use when the user mentions "bundle sync",
+  "sync wiki", "enrich knowledge bundles".
+---
+
+# LLW OKF SYNC
+
+The `llm-wiki` vault stores knowledge bundles in the `bundles/` directory, at the root of the vault. Each knowledge bundle is a folder that can contain 
+concepts, and other child bundles, every bundles folder contains a `bundle.md` file that marks the root of that bundle and contains the knowledge bundles metadata.
+This skill is used to sync all knowledge bundles across the vault, find outOfSync `bundle.md` files, and enrich them with its directory of contents. 
+
+## Quick Reference
+
+**Bundle Metadata/Frontmatter Fields:**
+
+| Field | Required? | Description |
+|-------|-----------|-------------|
+| `title` | YES | title of knowledge bundle (default to bundle folder name) |
+| `description` | Recommended | One-sentence summary of knowledge bundle |
+| `triggers` | Optional | YAML list for when to utilize knowledge base |
+| `parent` | YES(only if parent bundle exists) | for subdirectory groups in bundles to reference its parent bundle as a obsidian backlink  |
+
+
+parent backlink format: `[[path/to/parent/bundle|parent-bundle-title]]`
+
+**Concept Metadata/Frontmatter Fields:**
+
+| Field | Required? | Description |
+|-------|-----------|-------------|
+| `type` | YES | A short string identifying the kind of concept. Consumers use this for routing, filtering, and presentation. |
+| `title` | YES | title of the concept name (default to concept file name) |
+| `description` | Recommended | A single sentence summarizing the concept. |
+| `resource` | Optional | A URI that uniquely identifies the underlying asset the concept describes |
+| `tags` | Optinal | A YAML list of short strings for cross-cutting categorization.  |
+| `timestamp` | YES | ISO 8601 datetime of last meaningful change.  |
+
+
+**Example Folder Structure:**
+
+```
+path/to/bundles/some-backend/
+├── bundle.md                     # Bundle index that stores metadata of bundle title, description and triggers.
+├── log.md                        # Optional. Chronological history of updates.
+├── <concept>.md                  # A concept at the bundle root.
+└── database/                     # Subdirectories organize concepts into groups.
+    ├── bundle.md                 # subdirectory bundle index
+    └── <concept>.md
+```
+
+---
+
+## Bundle Syncing
+
+### Step 1: Sync Bundles
+
+```bash
+llw bundle sync
+```
+
+this command will output the Bundle Sync mainifest which contains summary all the bundles states and 3 filepaths that tracks the bundle syncing process.
+
+**Example Output:**
+```
+Bundle Sync Manifest
+
+════════════════════════════════════════════════════════════
+Summary:
+ ● Knowledge Bundles: 10 total, 5 outOfSync 
+ ● Pending Tasks: 3 in progress
+ ● Completed Tasks: 2 
+ ● Ignored Tasks: 0
+
+FilePaths
+────────────────────────────────────────────────────────────
+currentHashMap: <full_file_path_to_current_hash_map.json> 
+generatedHashMap: <full_file_path_to_generated_hash_map.json> 
+taskList: <full_file_path_to_task_list.md> 
+
+════════════════════════════════════════════════════════════
+```
+
+**FilePaths:**
+- currenHashMap: contains the last updated sync state of all bundles in the vault, or an empty json file (if it is the first time running the sync command)
+- generatedHashMap: contains the newly generated sync state of all bundles in the vault, with the timestamp of when it was generatted, and a Record of each bundleFolder and its bundleHash  
+- taskList: its an auto generated markdown file that contains a list of tasks which are the outOfSync bundles that needs to be updated. 
+
+Example hashMap json:
+```json
+{
+  "generatedTimestamp": "2026-07-25T01:55:40.519Z",
+  "bundles": {
+    "bundles/ecommerce": "c03b8d8330e03ec0fb3becc53c808db7",
+    "bundles/database": "103efa82af7bea22f8fe7449748d0bae",
+    "bundles/database/tables": "5cb8fc88c9abad625da2f4dfe80463ad",
+  }
+}
+```
+
+**Workflows:**
+1. if all tasks is completed/ignored, and there are 0 pending tasks, but there are still outOfSync bundles.
+  - copy the generatedHashMap file to the currentHashMap file, and delete the generatedHashMap file
+  - delete the taskList file, and rerun the sync command to check the output
+2. if there are 0 pending tasks and 0 outOfSync bundles, means there is nothing to sync, inform the user that the bundles sync is already up to date. 
+3. if there are pending tasks move on to step 2.
+
+---
+
+### Step 2: Updating Bundle.md files
+
+read through the taskList file for pending tasks
+```md
+1. [ ] bundles/some_bundle // pending
+2. [x] bundles/some_bundle/child_bundle // completed 
+3. [-] bundles/some_other_bundle // ignored 
+```
+
+**Guardrails:**
+- **DO NOT** update the `bundle.md` metadata/frontmatter fields, only update its contents
+
+example of the `bundle.md` frontmatter fields:
+```markdown
+---
+title: <title name> 
+description: <bundle_description>
+parent: <obsidian parent link> 
+triggers: [<trigger>, <trigger>, …]
+---
+```
+this **SHOULD NOT** be touched, updated, removed or changed in any capacity, only edit the contents below the frontmatter fields.
+
+- **MUST** proccess one task at a time, do not do multiple simultaneously
+- only need to read the `bundle.md` file thats being updated, do not need to read any other files, all required data i already provided with the command output mentioned below. 
+
+**Command to View Bundle**
+for each pending task in the tasklist, run teh following command to view the bundles data 
+
+```bash
+llw bundle view --bundle <bundle_folder_path>
+# example
+llw bundle view --bundle bundles/some_bundle
+```
+
+this command will output the bundles metadata, and the metadata of all its concepts and child bundles. This command output is what we use to derive the  
+bundles hash in the generatedHashMap file.
+
+**Important Fields:**
+- title: the title of the bundle (will be used for the `bundle.md` main header)
+- path: will be used with the vault path to read the `bundle.md` file `<vault_path>/<bundle_path>`
+- concepts: list of all concepts and its metadata in the bundle
+- childBundles: list of all child bundles and its metadata in the bundle
+
+using the current bundle `path` field along with the full path to the vault, read the bundle.md file and sees what needs to be updated.
+
+> [!NOTE] outOfSync bundles !== outdated bundles
+> just because a bundle is outOfSync, does not mean that the `bundle.md`file needs to be updated.
+> It could be that there was a minor change to its metadata field or one of its concepts or child bundles, that cause the bundle to have a different hash.
+> if thats the case, just mark the task as ignored and move on to the next task.
+
+
+#### Bundle.md Content Structure
+
+**Overview Structure**
+```md
+# <bundle-title> (capitalize first letter)
+
+...directory list of all concepts in the bundle.
+
+## Child Bundles
+...directory list of all child bundles in the bundle.
+```
+
+**Note:** for directory links description, do not reuse the `description` field, instead format it to make it more human readable
+as the `description` field is meant for agents to read but for, bundle sync, we want to make it more human readable for the user. 
+
+##### Updating/Adding Concepts Section
+
+based on the returned `concepts` and its metadata from the `llw bundle view` command, structure the concepts section in the `bundle.md` file as follows: 
+
+```markdown
+### Section / Group Heading
+
+- [Title 1](relative-url-1) - short description of concept 1
+- [Title 2](relative-url-2) - short description of concept 2
+
+### Another Section
+
+- [Title 3](relative-url-3) - short description of concept3 
+```
+
+group the concepts by their `type`, `description` and `tags` fields into logical sections. and remove any concepts links  that are no longer in the bundle.
+
+
+##### Updating/Adding Child Bundles Section
+
+based on the returned `childBundles` and its metadata from the `llw bundle view` command, structure the child bundles section in the `bundle.md` file as follows: 
+
+```markdown
+## Child Bundles 
+
+### Section / Group Heading
+
+- [Title 1](relative-url-1) - short description of bundle 1
+- [Title 2](relative-url-2) - short description of bundle 2
+
+### Another Section
+
+- [Title 3](relative-url-3) - short description of bundle3 
+```
+
+group the childBundles by `description` fields into logical sections. and remove any child bundle links that are no longer in the bundle.
+
+#### Marking Task as Completed
+- once the `bundle.md` file has been updated, edit the taskList file and mark the task  as completed, pending `[ ]` to completed `[x]`. 
+- move on to the next pending task in the taskList file and repeat the process until all tasks are completed or ignored.
+- once all tasks are no longer pending, move on to the next Step.
+
+---
+
+### Step 3: Finalizing Bundle Sync
+
+before finalizing the bundle sync, wait for user confirmation to continue. unless the user has `AUTO MODE` in their prompt, then automatically continue to finalize the bundle sync.
+
+**Response Format**
+```
+All Bundles have been Updated.
+Summary:
+- completed Tasks: <count> 
+- ignored Tasks: <count> 
+continue to finalize the bundle sync? (yes/no)
+```
+
+1. next replace the currentHashMap file with the generatedHashMap file, and delete the generatedHashMap file. (both schemas have to be the same)
+2. delete the taskList file.