@swarmvaultai/engine 0.1.32 → 0.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 SwarmVault
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -20,10 +20,12 @@ If you only want to use SwarmVault as a tool, install `@swarmvaultai/cli` instea
 ```ts
 import {
   addInput,
+  addManagedSource,
   benchmarkVault,
   compileVault,
   createMcpServer,
   createWebSearchAdapter,
+  deleteManagedSource,
   defaultVaultConfig,
   defaultVaultSchema,
   exploreVault,
@@ -40,6 +42,7 @@ import {
   getWebSearchAdapterForTask,
   lintVault,
   listGodNodes,
+  listManagedSourceRecords,
   listSchedules,
   loadVaultConfig,
   loadVaultSchema,
@@ -48,6 +51,7 @@ import {
   pushGraphNeo4j,
   queryGraphVault,
   queryVault,
+  reloadManagedSources,
   runWatchCycle,
   runSchedule,
   searchVault,
@@ -67,6 +71,7 @@ The engine also exports the main runtime types for providers, graph artifacts, p
 ```ts
 import {
   addInput,
+  addManagedSource,
   benchmarkVault,
   compileVault,
   exploreVault,
@@ -76,10 +81,12 @@ import {
   importInbox,
   initVault,
   installGitHooks,
+  listManagedSourceRecords,
   loadVaultSchemas,
   pushGraphNeo4j,
   queryGraphVault,
   queryVault,
+  reloadManagedSources,
   runWatchCycle,
   watchVault
 } from "@swarmvaultai/engine";
@@ -89,6 +96,8 @@ const rootDir = process.cwd();
 await initVault(rootDir, { obsidian: true });
 const schemas = await loadVaultSchemas(rootDir);
 console.log(schemas.root.path);
+const managed = await addManagedSource(rootDir, "https://github.com/karpathy/micrograd");
+console.log(managed.source.id);
 await addInput(rootDir, "https://arxiv.org/abs/2401.12345");
 await importInbox(rootDir);
 await compileVault(rootDir, {});
@@ -115,6 +124,8 @@ await pushGraphNeo4j(rootDir, {
 
 await runWatchCycle(rootDir, { repo: true });
 console.log(await getWatchStatus(rootDir));
+console.log(await listManagedSourceRecords(rootDir));
+await reloadManagedSources(rootDir, { all: true, compile: true });
 await installGitHooks(rootDir);
 
 const watcher = await watchVault(rootDir, { lint: true, repo: true });
@@ -170,15 +181,23 @@ This matters because many "OpenAI-compatible" backends only implement part of th
 
 ### Ingest
 
+- `addManagedSource(rootDir, input, { compile, brief, maxPages, maxDepth })` registers and syncs a recurring source, then optionally compiles and writes a source brief
+- `listManagedSourceRecords(rootDir)` lists registry-backed managed sources from `state/sources.json`
+- `reloadManagedSources(rootDir, { id, all, compile, brief, maxPages, maxDepth })` re-syncs one managed source or the full registry
+- `deleteManagedSource(rootDir, id)` removes a managed-source registry entry and transient sync state without deleting canonical vault artifacts
 - `ingestInput(rootDir, input, { includeAssets, maxAssetSize })` ingests a local file path or URL
 - `addInput(rootDir, input, { author, contributor })` captures supported URLs into normalized markdown before ingesting them, or falls back to generic URL ingest
 - `ingestDirectory(rootDir, inputDir, { repoRoot, include, exclude, maxFiles, gitignore, extractClasses })` recursively ingests a local directory as a repo-aware code/content source tree
 - `importInbox(rootDir, inputDir?)` recursively imports supported inbox files plus markdown and HTML browser-clipper style bundles
-- JavaScript, TypeScript, Python, Go, Rust, Java, C#, C, C++, PHP, Ruby, and PowerShell inputs are treated as code sources and compiled into both source pages and `wiki/code/` module pages
+- managed sources support local directories, public GitHub repo root URLs, and bounded same-domain docs hubs
+- registry data lives in `state/sources.json`, working state lives under `state/sources/<id>/`, and source briefs are written to `wiki/outputs/source-briefs/<id>.md`
+- JavaScript, TypeScript, Python, Go, Rust, Java, C#, C, C++, PHP, Ruby, PowerShell, Kotlin, and Scala inputs are treated as code sources and compiled into both source pages and `wiki/code/` module pages
+- `.rst` and `.rest` inputs are treated as first-class text sources with lightweight heading and directive normalization before analysis
 - code manifests can carry `repoRelativePath`, and compile writes `state/code-index.json` so local imports can resolve across an ingested repo tree
 - repo-aware manifests, graph nodes, and graph pages can also carry `sourceClass` so first-party, third-party, resource, and generated material can be filtered and reported separately
 - HTML and markdown URL ingests localize remote image references into `raw/assets/<sourceId>/` by default and rewrite the stored markdown to local relative paths
 - PDF and DOCX ingests now write extracted-text and metadata sidecars under `state/extracts/`, and image ingest keeps the same sidecar model for vision extraction
+- Tree-sitter-backed languages now verify runtime and grammar compatibility per language; failures stay local to the affected source and surface as diagnostics instead of aborting the whole compile
 
 ### Compile + Query
 
@@ -206,6 +225,7 @@ This matters because many "OpenAI-compatible" backends only implement part of th
 - `getWatchStatus(rootDir)` reads the latest watch-status artifact plus pending semantic refresh entries
 - `syncTrackedRepos(rootDir)` refreshes previously ingested repo roots, updates changed manifests, and removes deleted repo manifests
 - `syncTrackedReposForWatch(rootDir)` is the repo-watch sync path that defers non-code semantic refresh into `state/watch/`
+- large ingest and compile passes emit low-noise progress on TTYs, and report presentation rolls up tiny fragmented communities without mutating the canonical graph artifact
 - `installGitHooks(rootDir)`, `uninstallGitHooks(rootDir)`, and `getGitHookStatus(rootDir)` manage local `post-commit` and `post-checkout` hook blocks for the nearest git repository
 - `installAgent(rootDir, agent, { hook })` writes agent instructions and returns the primary `target`, all touched `targets`, and optional merge warnings for agents such as Aider
 - `lintVault(rootDir, options)` runs structural lint, optional deep lint, optional contradiction-only filtering through `{ conflicts: true }`, and optional web-augmented evidence gathering
@@ -245,6 +265,8 @@ Running the engine produces a local workspace with these main areas:
 - `wiki/projects/`: generated project rollups over canonical pages
 - `wiki/candidates/`: staged concept and entity pages awaiting confirmation on a later compile
 - `state/manifests/`: source manifests
+- `state/sources.json`: managed-source registry state
+- `state/sources/`: managed-source working state such as GitHub checkouts and crawl metadata
 - `state/extracts/`: extracted markdown plus JSON sidecars describing extractor kind, warnings, PDF page counts, and image-vision metadata
 - `state/analyses/`: model analysis output
 - `state/code-index.json`: repo-aware code module aliases and local resolution data