@jerryjiao/knowflow 0.3.0

package/CHANGELOG.md

@@ -0,0 +1,89 @@
+# Changelog
+
+All notable changes to KnowFlow are documented here. This project follows
+[Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+### Added
+
+- Bilingual documentation website (Astro + Starlight) under `site/`, deployed to
+  GitHub Pages via GitHub Actions. English landing at `/`, Chinese at `/zh/`,
+  docs under `/en/` and `/zh/` with browser-language auto-switch.
+
+## [0.3.0] - 2026-08-18
+
+### Added
+
+- First npm release, published as the scoped package `@jerryjiao/knowflow`
+  (npm considers the bare name too similar to the existing `know-flow`
+  package). Install with `npm install -g @jerryjiao/knowflow`; the CLI
+  command is still `knowflow`.
+- `knowflow tags` builds and prunes `tag/<name>.md` hub pages from `[[tag/<name>]]`
+  links, linking back to every tagged page. Re-running is idempotent.
+- `health.excludeOrphanDirs` configuration lists directories whose pages are
+  expected to be unreferenced (daily-sync feeds, inboxes) and should not count
+  as isolated pages.
+
+### Fixed
+
+- Link extraction missed every wikilink after the first one on a line, so
+  index-style pages that put dozens of links on a single line silently dropped
+  most of their references and produced mass false orphan reports.
+- Markdown link extraction captured a leading `(` into the target path.
+- `knowflow health` exited with code 0 even when it found issues, hiding
+  failures from cron and CI wrappers.
+
+### Changed
+
+- Both READMEs make the source checkout the primary install path until the
+  npm package is published, instead of an `npx` command that 404s.
+
+## [0.2.1] - 2026-08-10
+
+### Added
+
+- `knowflow fix [--dry-run]` repairs issues the health check surfaces: cleans empty
+  `[[entities/,]]` / `[[concepts/,]]` links, creates missing entity/concept pages,
+  pads files below the minimum size, and auto-links orphan pages to `index.md`.
+  `--dry-run` reports what would change without writing.
+
+### Fixed
+
+- `wiki-auto-fix.sh` (now backing `knowflow fix`) honored `--dry-run` for the
+  empty-link cleanup but still created, padded, and orphan-linked files in
+  dry-run mode because shell redirections ran before the dry-run guard. All
+  three write paths now short-circuit under `--dry-run` and `\n` escapes in the
+  pad suffix render as real newlines.
+
+## [0.2.0] - 2026-07-17
+
+### Added
+
+- `knowflow init [directory]` now creates a standalone project, its JSON configuration,
+  Wiki/raw/graph directories, starter index, and editable templates.
+- Core CLI tests using Node's built-in test runner and a GitHub Actions workflow.
+- npm publishing metadata and an explicit package file allowlist.
+- English and Simplified Chinese project guides, a runnable text-to-graph example,
+  contributor and security policies, and structured GitHub issue forms.
+
+### Changed
+
+- Commands resolve project paths from the nearest `.knowflowrc` and support custom
+  Wiki, raw, and graph locations.
+- External commands use argument arrays instead of interpolated shell strings.
+- Graph, ingestion, bookmark, health, and vector scripts use portable project paths.
+
+### Fixed
+
+- Fixed the undefined graph Wiki path and browser launcher in the CLI.
+- Fixed a syntax error and ESM compatibility in `enrich-wiki.js`.
+- Removed machine-specific paths from graph scripts.
+
+## [0.1.0] - 2026-04-28
+
+- Initial open-source release.
+
+[0.2.1]: https://github.com/jerryjiao/knowflow/compare/v0.2.0...v0.2.1
+[0.2.0]: https://github.com/jerryjiao/knowflow/compare/d0ff6f4...v0.2.0
+[0.1.0]: https://github.com/jerryjiao/knowflow/commit/d0ff6f4

package/CONTRIBUTING.md

@@ -0,0 +1,41 @@
+# Contributing to KnowFlow
+
+Thanks for helping make agent-native knowledge work more practical. Focused bug fixes, documentation improvements, new source adapters, templates, and reproducible workflow examples are all welcome.
+
+## Before opening a pull request
+
+1. Check existing issues and explain the user problem your change solves.
+2. Keep capture and Wiki synthesis conceptually separate: `ingest` saves raw material; a human or agent organizes structured pages.
+3. Preserve portable project paths and existing Markdown/configuration formats.
+4. Add or update tests for observable CLI behavior.
+5. Run the complete local verification suite:
+
+   ```bash
+   npm install
+   npm run check
+   npm test
+   npm pack --dry-run
+   ```
+
+## Project conventions
+
+- Node.js code uses ES modules unless a script has an explicit compatibility reason.
+- Python targets 3.10+ and should include type annotations for new public functions.
+- Shell scripts use `set -euo pipefail` and quote path/input variables.
+- Commit messages should follow Conventional Commits, such as `feat:`, `fix:`, or `docs:`.
+- Never commit API keys, `.env` files, captured private content, or machine-specific absolute paths.
+
+## Useful starting points
+
+| Goal | File or directory |
+| --- | --- |
+| Change CLI behavior | `bin/knowflow.js` |
+| Change capture behavior | `scripts/ingest.sh` |
+| Change graph generation | `scripts/graph_builder.py` |
+| Change Wiki templates | `templates/` |
+| Understand the architecture | `docs/architecture/system-architecture.md` |
+| Understand the data model | `docs/reference/data-model.md` |
+
+For the longer Chinese contributor guide, see [docs/contributing.md](docs/contributing.md). Please report vulnerabilities according to [SECURITY.md](SECURITY.md), not in a public issue.
+
+By contributing, you agree that your contribution will be licensed under the repository's [MIT License](LICENSE).

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 KnowFlow Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,177 @@
+# KnowFlow
+
+<p align="left"><img src="docs/assets/logo.png" width="120" alt="KnowFlow logo"></p>
+
+[English](README.md) | [简体中文](README.zh-CN.md) | [Website & docs](https://jerryjiao.github.io/knowflow/)
+
+[![CI](https://github.com/jerryjiao/knowflow/actions/workflows/ci.yml/badge.svg)](https://github.com/jerryjiao/knowflow/actions/workflows/ci.yml)
+[![Node.js 18+](https://img.shields.io/badge/Node.js-18%2B-339933?logo=node.js&logoColor=white)](https://nodejs.org/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+![KnowFlow — an agent-native Markdown knowledge workspace](docs/assets/knowflow-social-preview.png)
+
+> An agent-native Markdown knowledge workspace for capturing sources, organizing linked wiki pages, and exploring what you know.
+
+KnowFlow keeps the workflow inspectable: URLs and notes land in `raw/`, you or an AI agent turn them into structured Markdown pages, and local tools build a knowledge graph, check wiki health, and report project status. Optional semantic search is available when you configure an embedding API key.
+
+- **Own the knowledge layer** — plain Markdown, editable templates, and `[[wikilinks]]` instead of a closed database.
+- **Work with any agent** — keep capture deterministic, then use the agent or workflow you trust for synthesis.
+- **See structure, not just search results** — generate an interactive graph and catch broken or isolated pages.
+
+> **Demo:** [Run the text-to-graph walkthrough](examples/quickstart.md), then explore the generated graph in your browser.
+
+![KnowFlow interactive knowledge graph](docs/assets/knowflow-graph-demo.png)
+
+## Quick start
+
+Requires Node.js 18+, Python 3.10+, Bash, and curl.
+
+Install from npm — or [install from source](#install-from-source) if you prefer:
+
+```bash
+npm install -g @jerryjiao/knowflow
+knowflow init my-wiki
+cd my-wiki
+knowflow ingest "LLM Wikis turn notes into linked knowledge." --source text
+knowflow status
+knowflow graph --no-open
+```
+
+This creates a standalone project, captures one note in `raw/`, and builds a graph from the starter Wiki. **`ingest` does not automatically generate structured Wiki pages.** Organize captured material manually or with an AI agent/workflow before rebuilding the graph.
+
+One-off usage without a global install also works: `npx @jerryjiao/knowflow@latest <command>`.
+
+## The workflow
+
+```text
+URL or note
+    │
+    ▼
+raw/ Markdown ──► you, an AI agent, or a custom workflow
+                         │
+                         ▼
+                 linked Wiki pages
+                    │         │
+                    ▼         ▼
+             knowledge graph  optional semantic search
+```
+
+KnowFlow is inspired by [Andrej Karpathy's LLM Wiki](https://karpathy.github.io/llm-wiki/): knowledge becomes more useful when it is curated into durable, connected pages instead of being left in a pile of saved links.
+
+## What is included
+
+- Capture plain text and supported URLs as raw Markdown.
+- Initialize portable projects with JSON configuration and editable page templates.
+- Organize source, entity, concept, and comparison pages with `[[wikilinks]]`.
+- Generate `graph.html` and `graph.json` without an API key.
+- Check broken links, undersized files, and isolated pages.
+- Repair empty links, create missing pages, pad small files, and link orphans.
+- Query a previously built vector index with optional Zhipu AI embeddings.
+
+## Commands
+
+| Command | What it does |
+| --- | --- |
+| `knowflow init [directory]` | Create a standalone project; defaults to the current directory |
+| `knowflow ingest <url-or-text>` | Capture a URL or text in `raw/` |
+| `knowflow graph [--no-open]` | Generate `graph.html` and `graph.json` from Wiki pages |
+| `knowflow fix [--dry-run]` | Repair empty links, missing pages, small files, and orphans |
+| `knowflow health` | Check broken links, small files, and isolated pages |
+| `knowflow tags` | Build `tag/<name>.md` hub pages from `[[tag/<name>]]` links |
+| `knowflow status` | Show raw, Wiki, graph, vector-index, and API-key status |
+| `knowflow query <text>` | Query an existing vector index |
+
+KnowFlow searches upward from the current directory for the nearest `.knowflowrc`, so commands also work inside project subdirectories.
+
+## Install from source
+
+```bash
+git clone https://github.com/jerryjiao/knowflow.git
+cd knowflow
+npm install
+npm link
+knowflow init ../my-wiki
+```
+
+Running `init` again preserves existing configuration, the starter index, and customized templates.
+
+## Project layout
+
+```text
+my-wiki/
+├── .knowflowrc
+├── raw/                 # captured source material
+├── wiki/
+│   ├── index.md
+│   ├── sources/
+│   ├── entities/
+│   ├── concepts/
+│   └── comparisons/
+├── graph/               # generated graph.html and graph.json
+└── templates/           # editable Markdown templates
+```
+
+## Configuration
+
+`knowflow init` writes a JSON `.knowflowrc`. Relative paths resolve from the directory containing that file.
+
+```json
+{
+  "wiki": {
+    "root": "./wiki",
+    "rawDir": "./raw"
+  },
+  "graph": {
+    "output": "./graph/graph.html"
+  },
+  "health": {
+    "minFileSize": 100,
+    "excludeOrphanDirs": ["sources/"]
+  }
+}
+```
+
+`health.excludeOrphanDirs` lists directories whose pages are expected to be unreferenced (daily-sync feeds, inboxes) and should not count as isolated pages. `knowflow tags` regenerates every hub page under `wiki/tag/`, so re-running it after new tagged pages arrive is safe and idempotent.
+
+Graph generation, health checks, capture, and status do not require an API key. Semantic search requires `ZHIPUAI_API_KEY` in the project environment or a project-root `.env` file:
+
+```bash
+ZHIPUAI_API_KEY=your-key-here
+```
+
+The current CLI can query an existing vector index but does not yet expose a standalone `index` command. See [Current limitations](#current-limitations) before relying on search.
+
+## Current limitations
+
+- `ingest` captures raw material; structured Wiki synthesis is a separate human or agent step.
+- URL capture uses Jina Reader. YouTube and some logged-in platforms may also require `yt-dlp` or an authenticated browser workflow.
+- `query` needs a prebuilt vector index and a [Zhipu AI API key](https://open.bigmodel.cn/). Until an `index` CLI command is added, advanced users can run `node <knowflow-install>/scripts/vector-store.mjs build`.
+- `bookmark_sync.sh` depends on the optional third-party `ft` command.
+- Generated graph HTML loads vis-network from a CDN when opened.
+
+## Roadmap
+
+- [x] Standalone project initialization and portable paths
+- [x] Raw URL/text capture, Wiki health checks, and interactive graphs
+- [x] CLI tests and CI across supported Node.js versions
+- [ ] A first-class agent workflow from raw capture to reviewed Wiki pages
+- [ ] Incremental ingestion and duplicate-source detection
+- [ ] `knowflow index` and pluggable embedding providers
+- [ ] Extractor/plugin system and a local Web UI
+
+Ideas and focused pull requests are welcome. Start with the [contribution guide](CONTRIBUTING.md), run the [text-to-graph example](examples/quickstart.md), or propose a use case in [GitHub Issues](https://github.com/jerryjiao/knowflow/issues).
+
+## Development
+
+```bash
+npm install
+npm run check
+npm test
+npm pack --dry-run
+```
+
+See the [changelog](CHANGELOG.md) for release notes and the [security policy](SECURITY.md) for responsible disclosure.
+
+## License
+
+[MIT](LICENSE) © [Jerry Jiao](https://github.com/jerryjiao)

package/README.zh-CN.md

@@ -0,0 +1,176 @@
+# KnowFlow
+
+<p align="left"><img src="docs/assets/logo.png" width="120" alt="KnowFlow logo"></p>
+
+[English](README.md) | [简体中文](README.zh-CN.md) | [官网与文档](https://jerryjiao.github.io/knowflow/)
+
+[![CI](https://github.com/jerryjiao/knowflow/actions/workflows/ci.yml/badge.svg)](https://github.com/jerryjiao/knowflow/actions/workflows/ci.yml)
+[![Node.js 18+](https://img.shields.io/badge/Node.js-18%2B-339933?logo=node.js&logoColor=white)](https://nodejs.org/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+![KnowFlow — 面向 AI Agent 的 Markdown 知识工作区](docs/assets/knowflow-social-preview.png)
+
+> 一个面向 AI Agent 的 Markdown 知识工作区：采集来源、整理互联 Wiki，并看见知识之间的关系。
+
+KnowFlow 让整个流程保持透明、可检查：URL 和笔记先进入 `raw/`，再由你或 AI Agent 整理为结构化 Markdown 页面；本地工具负责生成知识图谱、检查 Wiki 健康度和显示项目状态。配置嵌入 API Key 后，还可启用语义检索。
+
+- **知识层归你所有** — 使用普通 Markdown、可编辑模板和 `[[wikilinks]]`，不被封闭数据库锁定。
+- **适配任意 Agent** — 采集保持确定性，知识提炼则交给你信任的 Agent 或工作流。
+- **看见结构，而不只是搜索结果** — 生成交互式图谱，并发现断链和孤立页面。
+
+> **Demo：** [运行从文本到图谱的完整演示](examples/quickstart.md)，然后在浏览器中探索生成的图谱。
+
+![KnowFlow 交互式知识图谱](docs/assets/knowflow-graph-demo.png)
+
+## 快速开始
+
+需要 Node.js 18+、Python 3.10+、Bash 和 curl。
+
+从 npm 安装——想从源码装也行（[见下文](#从源码安装)）：
+
+```bash
+npm install -g @jerryjiao/knowflow
+knowflow init my-wiki
+cd my-wiki
+knowflow ingest "LLM Wiki 把零散笔记整理成互联知识。" --source text
+knowflow status
+knowflow graph --no-open
+```
+
+这会创建一个独立项目、在 `raw/` 采集一条笔记，并根据初始 Wiki 生成图谱。**`ingest` 不会自动生成结构化 Wiki 页面。** 请先手动或通过 AI Agent / 自定义工作流整理原始素材，再重新生成图谱。
+
+不想全局安装也可以直接跑：`npx @jerryjiao/knowflow@latest <命令>`。
+
+## 工作流
+
+```text
+URL 或笔记
+    │
+    ▼
+raw/ Markdown ──► 你、AI Agent 或自定义工作流
+                         │
+                         ▼
+                   互联 Wiki 页面
+                    │         │
+                    ▼         ▼
+                 知识图谱    可选语义检索
+```
+
+KnowFlow 的灵感来自 [Andrej Karpathy 的 LLM Wiki](https://karpathy.github.io/llm-wiki/)：知识经过整理，成为持久、互联的页面，才会比一堆从未再打开的收藏更有价值。
+
+## 已包含的能力
+
+- 将纯文本和支持的 URL 采集为原始 Markdown。
+- 使用 JSON 配置和可编辑页面模板初始化可移植项目。
+- 通过 `[[wikilinks]]` 组织来源、实体、概念和对比页面。
+- 无需 API Key 即可生成 `graph.html` 和 `graph.json`。
+- 检查断链、过小文件和孤立页面。
+- 配置智谱 AI 嵌入后，查询已构建的向量索引。
+
+## 命令
+
+| 命令 | 作用 |
+| --- | --- |
+| `knowflow init [directory]` | 创建独立项目，默认使用当前目录 |
+| `knowflow ingest <url-or-text>` | 将 URL 或文本采集到 `raw/` |
+| `knowflow graph [--no-open]` | 根据 Wiki 页面生成 `graph.html` 和 `graph.json` |
+| `knowflow fix [--dry-run]` | 修复空链接、缺失页面、过小文件和孤儿页 |
+| `knowflow health` | 检查断链、小文件和孤立页面 |
+| `knowflow tags` | 根据 `[[tag/<名称>]]` 链接生成 `tag/<名称>.md` 聚合页 |
+| `knowflow status` | 显示原始素材、Wiki、图谱、向量索引和 API Key 状态 |
+| `knowflow query <text>` | 查询已有向量索引 |
+
+KnowFlow 会从当前目录向上查找最近的 `.knowflowrc`，因此也可以在项目子目录中运行命令。
+
+## 从源码安装
+
+```bash
+git clone https://github.com/jerryjiao/knowflow.git
+cd knowflow
+npm install
+npm link
+knowflow init ../my-wiki
+```
+
+再次运行 `init` 会保留已有配置、初始首页和已经自定义的模板。
+
+## 项目目录
+
+```text
+my-wiki/
+├── .knowflowrc
+├── raw/                 # 采集的原始素材
+├── wiki/
+│   ├── index.md
+│   ├── sources/
+│   ├── entities/
+│   ├── concepts/
+│   └── comparisons/
+├── graph/               # 生成的 graph.html 和 graph.json
+└── templates/           # 可编辑 Markdown 模板
+```
+
+## 配置
+
+`knowflow init` 会创建 JSON 格式的 `.knowflowrc`，所有相对路径都以该文件所在目录为基准解析。
+
+```json
+{
+  "wiki": {
+    "root": "./wiki",
+    "rawDir": "./raw"
+  },
+  "graph": {
+    "output": "./graph/graph.html"
+  },
+  "health": {
+    "minFileSize": 100,
+    "excludeOrphanDirs": ["sources/"]
+  }
+}
+```
+
+`health.excludeOrphanDirs` 列出预期不会被引用的目录（每日同步流水页、收件箱等），其中的页面不计入孤立页面。`knowflow tags` 每次都会全量重建 `wiki/tag/` 下的聚合页，新增带标签页面后重跑即可，幂等安全。
+
+图谱生成、健康检查、内容采集和状态查看均不需要 API Key。语义检索需要在项目环境变量或项目根目录 `.env` 中设置 `ZHIPUAI_API_KEY`：
+
+```bash
+ZHIPUAI_API_KEY=your-key-here
+```
+
+当前 CLI 可以查询已有向量索引，但尚未提供独立的 `index` 命令。依赖搜索功能前，请先阅读[当前限制](#当前限制)。
+
+## 当前限制
+
+- `ingest` 只采集原始素材；将其提炼为结构化 Wiki 是独立的人工或 Agent 步骤。
+- URL 采集依赖 Jina Reader；YouTube 和部分需要登录的平台可能还需要 `yt-dlp` 或带登录态的浏览器工作流。
+- `query` 需要预先构建的向量索引和[智谱 AI API Key](https://open.bigmodel.cn/)。在加入 `index` 命令前，高级用户可运行 `node <knowflow-install>/scripts/vector-store.mjs build`。
+- `bookmark_sync.sh` 依赖可选的第三方 `ft` 命令。
+- 生成的图谱 HTML 打开时会从 CDN 加载 vis-network。
+
+## 路线图
+
+- [x] 独立项目初始化和可移植路径
+- [x] URL / 文本原始采集、Wiki 健康检查和交互式图谱
+- [x] CLI 自动化测试与多 Node.js 版本 CI
+- [ ] 从原始素材到已审核 Wiki 页面的一等 Agent 工作流
+- [ ] 增量采集和重复来源检测
+- [ ] `knowflow index` 与可插拔嵌入提供商
+- [ ] 提取器 / 插件系统和本地 Web UI
+
+欢迎提交想法和范围清晰的 PR。你可以先阅读[贡献指南](CONTRIBUTING.md)、运行[从文本到图谱的示例](examples/quickstart.md)，或在 [GitHub Issues](https://github.com/jerryjiao/knowflow/issues) 中提出使用场景。
+
+## 开发
+
+```bash
+npm install
+npm run check
+npm test
+npm pack --dry-run
+```
+
+版本变化见[更新日志](CHANGELOG.md)，漏洞披露方式见[安全策略](SECURITY.md)。
+
+## 许可证
+
+[MIT](LICENSE) © [Jerry Jiao](https://github.com/jerryjiao)

package/SECURITY.md

@@ -0,0 +1,18 @@
+# Security Policy
+
+## Supported versions
+
+Security fixes are applied to the latest released minor version.
+
+| Version | Supported |
+| --- | --- |
+| 0.2.x | Yes |
+| < 0.2 | No |
+
+## Reporting a vulnerability
+
+Please do not open a public issue for a suspected vulnerability.
+
+Use GitHub's **Report a vulnerability** flow in the repository's [Security tab](https://github.com/jerryjiao/knowflow/security/advisories/new). Include the affected command or script, reproduction steps, impact, and any suggested mitigation. Remove API keys, private captured content, and other secrets from the report.
+
+You should receive an acknowledgement within seven days. We will validate the report, coordinate a fix and disclosure timeline, and credit you if requested.