@tryformation/querylight-cli 0.1.1 → 0.2.0

package/README.md

@@ -26,6 +26,8 @@ Run without installing globally:
 bunx @tryformation/querylight-cli init
 ```
 
+For agent and Python automation examples that use `bunx` and `uv`, see [`examples/skills/qli-bunx-uv/SKILL.md`](https://github.com/formation-res/querylight-cli/blob/main/examples/skills/qli-bunx-uv/SKILL.md).
+
 Install as a dependency:
 
 ```bash
@@ -38,6 +40,12 @@ Then run:
 npx qli --help
 ```
 
+If you prefer to avoid a local install, use:
+
+```bash
+bunx @tryformation/querylight-cli --help
+```
+
 ## Release
 
 Publish releases from semantic version tags such as `0.1.1`.
@@ -78,6 +86,8 @@ Initialize a workspace:
 qli init
 ```
 
+`qli init` creates the workspace config, enables dense and sparse retrieval for new workspaces, and pulls missing model assets when the runtime is available.
+
 Add a local docs directory:
 
 ```bash
@@ -87,7 +97,7 @@ qli source add directory ./docs --name "Local Docs" --tag docs
 Build the knowledge base:
 
 ```bash
-qli rebuild
+qli ingest
 ```
 
 Search it:
@@ -109,6 +119,18 @@ Generate retrieval context:
 qli context "How do I authenticate API requests?" --top-k 8
 ```
 
+## Example Skill: `qli` with `bunx` and `uv`
+
+The repository includes an example skill for running `qli` without a global install and calling it from Python with `uv`:
+
+- [`examples/skills/qli-bunx-uv/SKILL.md`](https://github.com/formation-res/querylight-cli/blob/main/examples/skills/qli-bunx-uv/SKILL.md)
+
+It covers:
+
+- running `qli` with `bunx @tryformation/querylight-cli`
+- using `--json` for automation and agents
+- calling `qli search` and `qli context` from Python with `subprocess`
+
 ## Example: Index `querylight.tryformation.com`
 
 This example uses a local linked build of `qli` to create a test knowledge base for the Querylight documentation website.
@@ -146,10 +168,13 @@ qli source add website https://querylight.tryformation.com \
   --tag docs
 ```
 
-5. Build the local index:
+`qli source add website` may also detect one blog or news feed and register it as a separate `rss` source. Use `--json` when another tool needs the full list of created sources.
+Use `qli source add page` for one page. Use `qli source add website` when you want crawling or feed detection.
+
+5. Ingest content and refresh the local index:
 
 ```bash
-qli rebuild
+qli ingest
 ```
 
 6. Inspect and query the result:
@@ -189,19 +214,30 @@ The default workspace is `.kb/`.
   logs/
 ```
 
+Vector model downloads are shared across workspaces under `~/.qli/models/` by default. `qli init` pulls missing model assets for enabled retrieval modes, so a new workspace is ready for vector indexing after setup.
+
 Use a custom workspace with:
 
 ```bash
 qli --workspace ./my-kb <command>
 ```
 
+Control the default remote concurrency in `config.yaml`:
+
+```yaml
+crawler:
+  maxConcurrentRequests: 5
+```
+
+Set `crawl.maxConcurrentRequests` on a website or RSS source when one source needs a different limit.
+
 ## Supported Sources
 
 Current source types:
 
 - `file`
 - `directory`
-- `url`
+- `page`
 - `website`
 - `rss`
 - `markdown`
@@ -224,10 +260,12 @@ All commands support:
 --workspace <path>
 --config <path>
 --json
+--silent
 --verbose
---quiet
 ```
 
+Long-running commands print progress to stderr by default. Use `--silent` to suppress progress output. Use `--json` when another tool needs stable structured output.
+
 ### Initialize
 
 ```bash
@@ -243,16 +281,25 @@ Add sources:
 ```bash
 qli source add file ./docs/guide.md --name "Guide"
 qli source add directory ./docs --name "Docs" --tag docs
-qli source add url https://example.com/docs/auth --name "Auth Page"
+qli source add page https://example.com/docs/auth --name "Auth Page"
 qli source add website https://example.com --name "Example Site" --max-depth 2 --max-pages 50
+qli source add website https://example.com --name "Example Site" --max-concurrent-requests 8
+qli source add website https://example.com --name "Example Site" --json
 qli source add rss https://example.com/feed.xml --name "Release Feed"
+qli source add rss https://example.com/feed.xml --name "Release Feed" --max-concurrent-requests 3
 ```
 
+`page` stores one page. `website` crawls a site and may detect one feed during registration.
+
+Website sources may detect one blog or news feed during registration. When qli can infer a shared article prefix such as `/blog/` or `/news/`, it adds that prefix to the website source excludes to reduce duplicate ingestion.
+Website and RSS sources default to `5` remote requests in flight per source. Override that in `config.yaml` or on the source.
+
 List and manage them:
 
 ```bash
 qli source list
 qli source config <source-id> --retention-days 30
+qli source config <source-id> --max-concurrent-requests 2
 qli source config <source-id> --name "Docs Feed" --tag rss docs
 qli source disable <source-id>
 qli source enable <source-id>
@@ -274,6 +321,8 @@ Or pull every model that is available on the current machine:
 qli models pull
 ```
 
+By default, `qli models pull` stores model assets in `~/.qli/models/` so multiple workspaces can reuse them.
+
 Then ask for documents related to an existing document id or URI:
 
 ```bash
@@ -287,9 +336,13 @@ qli related https://example.com/docs/auth
 qli ingest
 qli chunk
 qli index build
+qli rebuild --silent
 ```
 
-Run the full pipeline:
+`qli ingest` fetches source content, updates affected chunks, and refreshes the index.
+Remote website and RSS fetches run concurrently. By default qli allows `5` in-flight requests per source.
+
+Use `qli rebuild` when you want the explicit full pipeline command:
 
 ```bash
 qli rebuild
@@ -304,9 +357,9 @@ Search:
 ```bash
 qli search "pricing API limits"
 qli search "refund policy" --tag support --top-k 20
-qli search --source-type rss,url --since 2026-05-01 --has-publication-date --top-k 25
+qli search --source-type rss,page --since 2026-05-01 --has-publication-date --top-k 25
 qli search --source-name "Release Feed,Company Blog" --uri-prefix https://example.com/news,https://example.com/blog
-qli search --source-type rss,url --top-k 25 --json
+qli search --source-type rss,page --top-k 25 --json
 qli search "authentication" --json
 ```
 

package/dist/chunk/chunker.d.ts

@@ -1,9 +1,11 @@
+import { type ProgressHandler } from "../core/progress.js";
 import type { ChunkRecord, DocumentRecord, WorkspaceConfig } from "../types/models.js";
 export declare function buildChunksForDocument(document: DocumentRecord, markdown: string, config: WorkspaceConfig, prior?: Map<string, ChunkRecord>, seenAt?: string): ChunkRecord[];
-export declare function chunkDocuments({ workspacePath, sourceId, documentId }: {
+export declare function chunkDocuments({ workspacePath, sourceId, documentId, progress }: {
     workspacePath: string;
     sourceId?: string;
     documentId?: string;
+    progress?: ProgressHandler;
 }): Promise<{
     chunksWritten: number;
 }>;