sdtk-wiki-kit 0.1.0 → 0.1.2

package/README.md

@@ -28,6 +28,7 @@ Implemented in the Foundation/Beta package:
 | Capability | Command |
 |---|---|
 | Initialize SDTK-WIKI workspace | `sdtk-wiki init` |
+| Build a personal-brain without timestamp variables | `sdtk-wiki ingest`, `sdtk-wiki compile --mode safe --apply`, `sdtk-wiki query` |
 | Build graph, viewer, generated pages, and provenance | `sdtk-wiki atlas build` |
 | Open local viewer | `sdtk-wiki atlas open` |
 | Watch markdown sources and rebuild | `sdtk-wiki atlas watch` |
@@ -36,13 +37,18 @@ Implemented in the Foundation/Beta package:
 | Run non-destructive wiki lint | `sdtk-wiki lint` |
 | Run stale-page prune dry-run report | `sdtk-wiki wiki prune --dry-run` |
 | Generate local discovery plan from gap evidence | `sdtk-wiki wiki discover --plan` |
-| Generate compile dry-run preview from a local plan | `sdtk-wiki wiki compile --dry-run` |
-| Ask grounded questions over built graph | `sdtk-wiki ask` |
-| Save one redacted query record after successful Ask | `sdtk-wiki ask --save-query` |
+| Generate local discovery plan with beginner facade | `sdtk-wiki discover --plan` |
+| Run report-first maintenance cycle | `sdtk-wiki maintain --mode safe` |
+| Generate semantic extraction dry-run report | `sdtk-wiki wiki extract --dry-run` |
+| Generate compile dry-run preview and JSON sidecar | `sdtk-wiki wiki compile --dry-run` |
+| Apply an approved compile JSON sidecar | `sdtk-wiki wiki compile --apply --yes` |
+| Search generated personal-brain pages locally | `sdtk-wiki search` |
+| Query generated personal-brain pages locally | `sdtk-wiki query` |
+| Ask grounded questions over built graph | `sdtk-wiki ask` with `wiki.ask` entitlement/runtime preconditions |
+| Save one redacted query record after successful Ask | `sdtk-wiki ask --save-query` with `wiki.ask` entitlement/runtime preconditions |
 
 Not implemented in the Foundation/Beta runtime:
 
-- `sdtk-wiki wiki compile --apply`
 - automatic web discovery or web fetch
 - automatic source ingest from the web
 - destructive prune/delete/archive
@@ -60,6 +66,7 @@ Not implemented in the Foundation/Beta runtime:
 | `.sdtk/wiki/raw` | metadata-only raw/source registry |
 | `.sdtk/wiki/provenance` | source/build/ingest provenance |
 | `.sdtk/wiki/reports` | lint, prune, discover, and compile preview reports |
+| `.sdtk/wiki/personal-brain` | generated semantic personal-brain pages from explicit apply |
 | `.sdtk/wiki/queries` | opt-in redacted Ask query records |
 | `.sdtk/atlas` | legacy Atlas compatibility output, readable only |
 
@@ -88,6 +95,41 @@ Open the viewer:
 sdtk-wiki atlas open --project-path .
 ```
 
+## Simple Personal-Brain Flow
+
+Use this when you want the shortest local second-brain path and do not need to
+inspect timestamped JSON files:
+
+```powershell
+sdtk-wiki init --no-open
+sdtk-wiki ingest .\source-md
+sdtk-wiki compile --mode safe --apply
+sdtk-wiki query "multi-agent"
+sdtk-wiki lint
+sdtk-wiki discover --plan
+sdtk-wiki maintain --mode safe
+```
+
+Behavior:
+
+- `ingest` runs local semantic extraction and writes a report.
+- `compile --mode safe --apply` creates or refreshes the compile preview/JSON
+  sidecar internally, then applies the sidecar contract.
+- `query` is deterministic local search over `.sdtk/wiki/personal-brain`; it
+  does not use premium Ask, LLM/RAG, web fetch, or query history.
+- `discover` and `maintain` are report-first and do not apply, delete, archive,
+  fetch web sources, or mutate `.sdtk/atlas`.
+
+Interactive viewer flow:
+
+```powershell
+sdtk-wiki init
+```
+
+Bare `sdtk-wiki init` may open/start the viewer and keep the terminal session
+active. Use `sdtk-wiki init --no-open` for automation, CI, and local
+non-interactive validation.
+
 ## Command Reference
 
 ### Init
@@ -128,6 +170,20 @@ Safety:
 - no generated page updates
 - no graph/viewer rebuild side effects
 
+### Beginner Ingest Operation
+
+```powershell
+sdtk-wiki ingest <source-root> [--project-path <path>]
+sdtk-wiki ingest --source-root <source-root> [--project-path <path>]
+```
+
+Runs semantic extraction over a local Markdown source root and writes the latest
+`semantic-extraction-dry-run-*.json` report under `.sdtk/wiki/reports`.
+
+This is the beginner-friendly facade for `sdtk-wiki wiki extract --dry-run`.
+It does not mutate source files, personal-brain pages, raw/provenance state,
+graph outputs, web sources, Ask state, or `.sdtk/atlas`.
+
 ### Lint
 
 ```powershell
@@ -164,14 +220,49 @@ Safety:
 - no compile/apply
 - no prune/delete/archive
 
+Beginner facade:
+
+```powershell
+sdtk-wiki discover --plan [--project-path <path>]
+```
+
+This uses the same local-only discovery plan behavior and report-first safety
+boundary.
+
+### Semantic Extraction Dry-Run
+
+```powershell
+sdtk-wiki wiki extract --project-path <path> --source-root <path> --dry-run
+```
+
+Reads local Markdown sources and writes a semantic extraction JSON report under
+`.sdtk/wiki/reports`. The report can identify local source records, GitHub
+tool candidates, concept candidates, relations, comparisons, syntheses, and
+source-quality findings.
+
+Safety:
+
+- local source roots only
+- no web fetch
+- no page generation
+- no graph/viewer rebuild side effects
+- no raw/provenance mutation
+- no `.sdtk/atlas` mutation
+
 ### Compile Dry-Run Preview
 
 ```powershell
 sdtk-wiki wiki compile --plan <path> --project-path <path> --dry-run
 ```
 
-Reads a local markdown or JSON compile plan and writes a preview report under
-`.sdtk/wiki/reports`.
+Reads a local structured markdown plan, JSON operation plan, or
+`sdtk_wiki_semantic_extraction` JSON report and writes both:
+
+- `.sdtk/wiki/reports/compile-dry-run-preview-YYYY-MM-DD.md`
+- `.sdtk/wiki/reports/compile-apply-plan-YYYY-MM-DD.json`
+
+The markdown report is for human review. The JSON sidecar is the only supported
+source of truth for explicit apply.
 
 Supported operation types:
 
@@ -180,9 +271,68 @@ Supported operation types:
 - `add_relation`
 - `add_source_ref`
 
-Unknown operation types are reported as `unsupported_operation`. The current
-runtime has no `--apply` behavior and does not modify wiki pages, raw sources,
-provenance, or `.sdtk/atlas`.
+Unknown operation types are reported as `unsupported_operation`. Dry-run does
+not modify wiki pages, raw sources, provenance, or `.sdtk/atlas`.
+
+### Beginner Compile Operation
+
+```powershell
+sdtk-wiki compile --mode safe [--project-path <path>]
+sdtk-wiki compile --mode safe --apply [--project-path <path>]
+```
+
+The beginner facade hides timestamped JSON paths:
+
+- without `--apply`, it finds the latest semantic extraction report and writes
+  the compile preview plus JSON sidecar
+- with `--apply`, it finds the latest sidecar or creates one from the latest
+  extraction report, then applies through the same sidecar-only contract
+
+`safe` is the only R1 mode. Auto mode, destructive cleanup, web fetch, Ask, raw
+mutation, provenance mutation, source mutation, and `.sdtk/atlas` mutation are
+not performed.
+
+### Compile Apply
+
+```powershell
+sdtk-wiki wiki compile --plan <compile-apply-plan-json> --project-path <path> --apply --yes
+```
+
+Applies only a `record_type: "sdtk_wiki_compile_apply_plan"` JSON sidecar
+generated by compile dry-run. Markdown plans and raw semantic extraction JSON
+are rejected for apply.
+
+Apply behavior:
+
+- requires `--apply --yes`
+- writes only under `.sdtk/wiki/personal-brain`
+- create-only or same-content no-op
+- no overwrite with different content
+- no delete, archive, rewrite, or reorder
+- no raw/provenance descriptor mutation
+- no `.sdtk/atlas` mutation
+
+### Local Search
+
+```powershell
+sdtk-wiki search --project-path <path> "<query>"
+```
+
+Searches generated personal-brain Markdown pages under
+`.sdtk/wiki/personal-brain/**/*.md`.
+
+Search is deterministic, read-only, and non-premium. It does not require
+`wiki.ask` entitlement, does not call an LLM/RAG runtime, does not write query
+history, and does not mutate project files.
+
+Beginner query facade:
+
+```powershell
+sdtk-wiki query --project-path <path> "<query>"
+```
+
+`query` is the simple local search command. It is not premium Ask and does not
+require `wiki.ask`.
 
 ### Ask
 
@@ -190,7 +340,9 @@ provenance, or `.sdtk/atlas`.
 sdtk-wiki ask --question "<text>" [--project-path <path>] [--json] [--source <id-or-path>] [--max-sources <n>] [--save-query]
 ```
 
-Native `sdtk-wiki ask` is the canonical Q&A command for capability `wiki.ask`.
+Native `sdtk-wiki ask` is implemented as the canonical Q&A command for
+capability `wiki.ask`, but it is not a free local search command. It requires
+valid `wiki.ask` entitlement and runtime preconditions.
 
 Preconditions:
 
@@ -227,8 +379,8 @@ Run report-first maintenance checks:
 
 ```powershell
 sdtk-wiki lint --project-path .
-sdtk-wiki wiki prune --dry-run --project-path .
-sdtk-wiki wiki discover --plan --project-path .
+sdtk-wiki discover --plan --project-path .
+sdtk-wiki maintain --mode safe --project-path .
 ```
 
 Preview a compile plan without applying it:
@@ -237,6 +389,23 @@ Preview a compile plan without applying it:
 sdtk-wiki wiki compile --plan <local-plan.md-or-json> --project-path . --dry-run
 ```
 
+Build a personal-brain from local Markdown sources and search it:
+
+```powershell
+sdtk-wiki ingest docs
+sdtk-wiki compile --mode safe --apply
+sdtk-wiki query "multi-agent"
+```
+
+Advanced/audit workflow with explicit report files:
+
+```powershell
+sdtk-wiki wiki extract --project-path . --source-root docs --dry-run
+sdtk-wiki wiki compile --project-path . --plan .sdtk/wiki/reports/semantic-extraction-dry-run-<stamp>.json --dry-run
+sdtk-wiki wiki compile --project-path . --plan .sdtk/wiki/reports/compile-apply-plan-<date>.json --apply --yes
+sdtk-wiki search --project-path . "multi-agent"
+```
+
 Ask and save an opt-in redacted query record:
 
 ```powershell
@@ -244,6 +413,10 @@ sdtk-wiki atlas build --project-path .
 sdtk-wiki ask --project-path . --question "Which docs describe the deployment path?" --save-query
 ```
 
+This flow requires valid `wiki.ask` entitlement/runtime preconditions. Use
+`sdtk-wiki search` for non-premium local validation of generated
+personal-brain pages.
+
 ## Foundation/Beta Boundaries
 
 This release is local-first and report-first. It is a foundation for a
@@ -252,10 +425,10 @@ second-brain workflow, not a fully autonomous second brain.
 Do not claim the Foundation/Beta runtime includes:
 
 - web fetch/discover
-- compile `--apply`
 - destructive prune/delete/archive
 - query list/show/delete
 - default full prompt/full answer query persistence
+- premium Ask without valid `wiki.ask` entitlement/runtime preconditions
 - `.sdtk/atlas` as canonical storage
 
 See `products/sdtk-wiki/governance/SDTK_WIKI_USAGE_GUIDE.md` for the fuller

package/assets/atlas/build_atlas.py

@@ -147,20 +147,48 @@ def _assert_inside(base: Path, target: Path) -> None:
         raise ValueError(f"Refusing to write outside SDTK-WIKI workspace: {resolved_target}")
 
 
-def _is_excluded(
-    path: Path,
-    root: Path,
-    exclude_frags: list[str],
-) -> bool:
-    try:
-        rel = path.relative_to(root).as_posix().lower()
-    except ValueError:
-        rel = path.as_posix().lower()
-    for frag in exclude_frags:
-        norm_frag = frag.replace("\\", "/").lower()
-        if norm_frag in rel:
-            return True
-    return False
+def _is_excluded(
+    path: Path,
+    root: Path,
+    exclude_frags: list[str],
+) -> bool:
+    return _match_exclude(path=path, root=root, exclude_frags=exclude_frags) is not None
+
+
+def _display_scan_path(path: Path, root: Path) -> str:
+    try:
+        return path.relative_to(root).as_posix()
+    except ValueError:
+        return path.as_posix()
+
+
+def _normalise_exclude_fragment(frag: str) -> list[str]:
+    norm_frag = frag.replace("\\", "/").strip("/").lower()
+    return [part for part in norm_frag.split("/") if part and part != "."]
+
+
+def _match_exclude(
+    path: Path,
+    root: Path,
+    exclude_frags: list[str],
+) -> str | None:
+    rel = _display_scan_path(path, root).lower()
+    rel_parts = [part for part in rel.split("/") if part and part != "."]
+
+    for frag in exclude_frags:
+        frag_parts = _normalise_exclude_fragment(frag)
+        if not frag_parts:
+            continue
+        if len(frag_parts) == 1:
+            if frag_parts[0] in rel_parts:
+                return frag
+            continue
+
+        for idx in range(0, len(rel_parts) - len(frag_parts) + 1):
+            if rel_parts[idx : idx + len(frag_parts)] == frag_parts:
+                return frag
+
+    return None
 
 
 def _extract_title(text: str) -> str:
@@ -322,9 +350,9 @@ def _compute_file_hash(md_file: Path) -> str:
     return hashlib.sha256(content).hexdigest()
 
 
-def _parse_doc_record(md_file: Path, root: Path) -> dict[str, Any]:
-    rel = md_file.relative_to(root).as_posix()
-    text = md_file.read_text(encoding="utf-8", errors="replace")
+def _parse_doc_record(md_file: Path, root: Path) -> dict[str, Any]:
+    rel = _display_scan_path(md_file, root)
+    text = md_file.read_text(encoding="utf-8", errors="replace")
     frontmatter_fields, body_text = _parse_frontmatter(text)
     title = str(
         frontmatter_fields.get("title")
@@ -363,39 +391,70 @@ def _parse_doc_record(md_file: Path, root: Path) -> dict[str, Any]:
     }
 
 
-def list_indexable_markdown_files(
-    root: Path,
-    scan_roots: list[Path],
-    exclude_frags: list[str],
-) -> list[Path]:
-    files: list[Path] = []
-    seen_paths: set[str] = set()
-
-    for scan_root in scan_roots:
-        if not scan_root.exists():
-            print(f"[atlas] Warning: scan root does not exist, skipping: {scan_root}", file=sys.stderr)
-            continue
+def list_indexable_markdown_files(
+    root: Path,
+    scan_roots: list[Path],
+    exclude_frags: list[str],
+) -> list[Path]:
+    return collect_indexable_markdown_files(root, scan_roots, exclude_frags)["files"]
+
+
+def collect_indexable_markdown_files(
+    root: Path,
+    scan_roots: list[Path],
+    exclude_frags: list[str],
+) -> dict[str, Any]:
+    files: list[Path] = []
+    seen_paths: set[str] = set()
+    skipped_files: list[dict[str, str]] = []
+    scanned_count = 0
+
+    for scan_root in scan_roots:
+        if not scan_root.exists():
+            print(f"[atlas] Warning: scan root does not exist, skipping: {scan_root}", file=sys.stderr)
+            continue
         if scan_root.is_file() and scan_root.suffix.lower() == ".md":
             candidates = [scan_root]
         elif scan_root.is_dir():
             candidates = [p for p in sorted(scan_root.rglob("*.md")) if p.is_file()]
         else:
-            candidates = []
-
-        for md_file in candidates:
-            if _is_excluded(md_file, root=root, exclude_frags=exclude_frags):
-                continue
-            try:
-                rel = md_file.relative_to(root).as_posix()
-            except ValueError:
-                rel = md_file.as_posix()
-            if rel in seen_paths:
-                continue
-            seen_paths.add(rel)
-            files.append(md_file)
-
-    files.sort(key=lambda p: p.as_posix())
-    return files
+            candidates = []
+
+        for md_file in candidates:
+            scanned_count += 1
+            matched_exclude = _match_exclude(md_file, root=root, exclude_frags=exclude_frags)
+            display_path = _display_scan_path(md_file, root)
+            if matched_exclude is not None:
+                skipped_files.append(
+                    {
+                        "path": display_path,
+                        "reason": f"exclude:{matched_exclude}",
+                    }
+                )
+                continue
+            try:
+                rel = md_file.relative_to(root).as_posix()
+            except ValueError:
+                rel = md_file.as_posix()
+            if rel in seen_paths:
+                skipped_files.append(
+                    {
+                        "path": display_path,
+                        "reason": "duplicate_scan_root",
+                    }
+                )
+                continue
+            seen_paths.add(rel)
+            files.append(md_file)
+
+    files.sort(key=lambda p: p.as_posix())
+    return {
+        "files": files,
+        "scanned_count": scanned_count,
+        "indexed_count": len(files),
+        "skipped_count": len(skipped_files),
+        "skipped_files": skipped_files,
+    }
 
 
 # ---------------------------------------------------------------------------
@@ -639,16 +698,17 @@ def write_wiki_pages_and_provenance(
     }
 
 
-def build_docs_incremental(
-    root: Path,
-    atlas_dir: Path,
-    generated: str,
-    scan_roots: list[Path],
-    exclude_frags: list[str],
-) -> tuple[list[dict[str, Any]], dict[str, Any], dict[str, int]]:
-    prior_state = load_atlas_state(atlas_dir)
-    prior_documents = prior_state.get("documents", {})
-    current_files = list_indexable_markdown_files(root, scan_roots, exclude_frags)
+def build_docs_incremental(
+    root: Path,
+    atlas_dir: Path,
+    generated: str,
+    scan_roots: list[Path],
+    exclude_frags: list[str],
+) -> tuple[list[dict[str, Any]], dict[str, Any], dict[str, Any]]:
+    prior_state = load_atlas_state(atlas_dir)
+    prior_documents = prior_state.get("documents", {})
+    scan_result = collect_indexable_markdown_files(root, scan_roots, exclude_frags)
+    current_files = scan_result["files"]
 
     current_rel_paths = {}
     for md_file in current_files:
@@ -710,12 +770,16 @@ def build_docs_incremental(
         "generated": generated,
         "documents": next_documents,
     }
-    build_stats = {
-        "discovered_count": len(current_rel_paths),
-        "reused_count": reused_count,
-        "reparsed_count": reparsed_count,
-        "removed_count": removed_count,
-    }
+    build_stats = {
+        "discovered_count": len(current_rel_paths),
+        "scanned_count": scan_result["scanned_count"],
+        "indexed_count": len(current_rel_paths),
+        "skipped_count": scan_result["skipped_count"],
+        "skipped_files": scan_result["skipped_files"],
+        "reused_count": reused_count,
+        "reparsed_count": reparsed_count,
+        "removed_count": removed_count,
+    }
     return docs, next_state, build_stats
 
 
@@ -814,11 +878,11 @@ def build_graph(docs: list[dict[str, Any]]) -> dict[str, Any]:
 # ---------------------------------------------------------------------------
 # Summary markdown
 # ---------------------------------------------------------------------------
-def build_summary(
+def build_summary(
     docs: list[dict[str, Any]],
     graph: dict[str, Any],
     generated: str,
-    stats: dict[str, int] | None,
+    stats: dict[str, Any] | None,
     root: Path,
     scan_roots: list[Path],
     exclude_frags: list[str],
@@ -848,16 +912,30 @@ def build_summary(
     for fam, cnt in sorted(family_counts.items(), key=lambda x: -x[1]):
         lines.append(f"| {fam} | {cnt} |")
 
-    if stats is not None:
-        lines += [
-            "",
-            "## Incremental Build",
-            "",
-            f"Discovered markdown docs: {stats['discovered_count']}",
-            f"Reused cached docs: {stats['reused_count']}",
-            f"Reparsed docs: {stats['reparsed_count']}",
-            f"Removed stale docs: {stats['removed_count']}",
-        ]
+    if stats is not None:
+        lines += [
+            "",
+            "## Incremental Build",
+            "",
+            f"Discovered markdown docs: {stats['discovered_count']}",
+            f"Scanned markdown candidates: {stats.get('scanned_count', stats['discovered_count'])}",
+            f"Indexed markdown docs: {stats.get('indexed_count', stats['discovered_count'])}",
+            f"Skipped markdown docs: {stats.get('skipped_count', 0)}",
+            f"Reused cached docs: {stats['reused_count']}",
+            f"Reparsed docs: {stats['reparsed_count']}",
+            f"Removed stale docs: {stats['removed_count']}",
+        ]
+        skipped_files = stats.get("skipped_files") or []
+        if skipped_files:
+            lines += [
+                "",
+                "## Skipped Markdown Files",
+                "",
+                "| Path | Reason |",
+                "|------|--------|",
+            ]
+            for skipped in skipped_files:
+                lines.append(f"| {skipped['path']} | {skipped['reason']} |")
 
     lines += [
         "",
@@ -959,12 +1037,19 @@ def build_atlas(
         scan_roots=roots,
         exclude_frags=frags,
     )
-    print(f"[atlas] Indexed {len(docs)} documents.")
-    if verbose:
-        print(
-            f"[atlas] Incremental build: reused {stats['reused_count']} cached, "
-            f"reparsed {stats['reparsed_count']}, removed {stats['removed_count']}."
-        )
+    print(f"[atlas] Indexed {len(docs)} documents.")
+    print(
+        f"[atlas] Scan coverage: scanned {stats.get('scanned_count', len(docs))}, "
+        f"indexed {stats.get('indexed_count', len(docs))}, "
+        f"skipped {stats.get('skipped_count', 0)}."
+    )
+    if verbose:
+        print(
+            f"[atlas] Incremental build: reused {stats['reused_count']} cached, "
+            f"reparsed {stats['reparsed_count']}, removed {stats['removed_count']}."
+        )
+        for skipped in stats.get("skipped_files", []):
+            print(f"[atlas] Skipped markdown: {skipped['path']} ({skipped['reason']})")
 
     print("[atlas] Building graph...")
     graph = build_graph(docs)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sdtk-wiki-kit",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "description": "Project-local wiki and knowledge graph toolkit for SDTK workspaces.",
   "bin": {
     "sdtk-wiki": "bin/sdtk-wiki.js"