TalkPipe 1.0.1__tar.gz → 1.0.2__tar.gz

{talkpipe-1.0.1 → talkpipe-1.0.2}/.github/workflows/ci-cd.yml

@@ -353,8 +353,11 @@ jobs:
     # blocked behind a skipped job everywhere else. The container image is
     # published independently by that job on github.com.
     needs: [test, lint, package, security-scan]
-    if: github.event_name == 'release' && github.event.action == 'published'
-    
+    # Version tags only: publishing a release under any other tag would upload
+    # a distribution whose setuptools_scm version came from a tag that is not
+    # one.
+    if: github.event_name == 'release' && github.event.action == 'published' && startsWith(github.event.release.tag_name, 'v')
+
     steps:
     - uses: actions/checkout@v7
       with:
@@ -380,4 +383,4 @@ jobs:
       env:
         TWINE_USERNAME: __token__
         TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
-      run: twine upload dist/*
+      run: twine upload dist/*

{talkpipe-1.0.1 → talkpipe-1.0.2}/AGENTS.md

@@ -12,7 +12,7 @@
 - `run_doc_examples` — extract and run all doc examples (alternative to pytest; requires full_network for LLM examples)
 
 ## Project Overview
-TalkPipe is a Python toolkit that makes it easy to create, test, and deploy workflows that integrate Generative AI with your existing tools and data sources. TalkPipe treats LLMs as one tool in your arsenal - letting you build practical solutions that combine AI with data processing, file handling, and more.  
+TalkPipe is a Python toolkit for creating, testing, and deploying workflows that integrate Generative AI with your existing tools and data sources. It treats LLMs as one tool among many, so solutions combine AI with data processing, file handling, and more.
 
 - **Chat with LLMs** - Create multi-turn conversations with OpenAI, Ollama, or Anthropic models in just 2 lines of code
 - **Process Documents** - Extract text from PDFs, analyze research papers, score content relevance

{talkpipe-1.0.1 → talkpipe-1.0.2}/CHANGELOG.md

@@ -1,5 +1,138 @@
 # Changelog
 
+## Unreleased
+
+## 1.0.2 (2026-09-25)
+
+- **`max_tokens` caps the length of an LLM response.** `llmPrompt` and the
+  guided-generation segments built on it (`llmScore`, `llmExtractTerms`,
+  `llmBinaryAnswer`) accept `max_tokens`, as do the Ollama/OpenAI/Anthropic
+  prompt adapters: the most tokens the model may generate per response
+  (output only; prompt tokens do not count). The Ollama adapter's chat path
+  sent only `temperature`, so a model that never emitted a stop token
+  generated until its context window filled; `timeout` fails the pipeline
+  in that case, while `max_tokens` bounds the item. It maps to `num_predict`
+  on Ollama, `max_output_tokens` on the OpenAI Responses API, and
+  `max_tokens` on Anthropic, where it replaces the adapter's 4096 (still the
+  value when unset). The default is `None`, which leaves every request
+  exactly as it was; there is no global config key, the option is
+  per-segment. On reasoning/thinking models the thinking tokens count
+  against the cap, and a truncated response from a guided-generation segment
+  fails schema validation rather than returning a partial result. Prompt
+  adapters that take no `max_tokens` (such as `eliza`, or third-party
+  adapters that predate it) keep working and ignore it.
+- **Removed: the TalkPipe App Center.** The single-file installer previewed in
+  the 1.0.2 betas (`appcenter/`, its release assets, and the rolling
+  `experimental` GitHub release that served them) is gone; it never shipped in
+  a full release. The TalkPipe applications install directly with
+  `uv tool install` — the README lists them with the commands. Textual and
+  pytest-asyncio left the `dev` extra. The `experimental` *container* image
+  tag for pre-releases is unaffected.
+- **Deprecated: omitting the `|` between an input source and the first
+  segment.** `INPUT FROM echo[data="1,2"] print` has always parsed as though
+  the pipe were there — an accident of the grammar, and an inconsistent one,
+  since every *later* pipe was required and `INPUT FROM echo[data="1,2"] print
+  print` was a syntax error. Such scripts still compile and run unchanged, but
+  compiling one now emits a `DeprecationWarning` naming the line and column of
+  the segment that needs the pipe, logs the same message (so it is visible to
+  anyone running `chatterlang_script`, where Python hides deprecation warnings
+  by default), and marks the spot in the workbench editor. The spelling will
+  be a syntax error in TalkPipe 2.0. Nothing else changed: a pipeline with no
+  input source of its own — a fork branch, a fork consumer
+  (`fork_name -> print`), or a fragment such as `| print` — still begins with
+  a bare segment, and spellings that were errors before are still errors.
+- **Documentation: TalkPipe is provider-neutral, and the docs now say so.**
+  Several places read as though Ollama were required — prerequisite lists that
+  named only Ollama, a quickstart whose LLM section began "requires
+  `talkpipe[ollama]` and Ollama running", RAG prerequisites that asked for an
+  Ollama completion model, and an installation note that suggested provider
+  API keys could live in `~/.talkpipe.toml`. The model and source guide gained
+  an authoritative **LLM providers** section — the four built-in providers
+  (Ollama, OpenAI, and Anthropic for chat and vision; Ollama, OpenAI, and
+  in-process model2vec for embeddings, plus the scripted eliza fallback), what
+  each one needs, how to select it per segment or by default, and how to
+  register an adapter of your own — and the README, quickstart, docs index,
+  RAG guide, tutorials, and command references link to it instead of repeating
+  it. Existing examples still use Ollama; they now say so and show the
+  one-parameter switch. Segment docstrings, `serverag` /
+  `makevectordatabase` / `chatterlang_workbench` help text, and the workbench's
+  example scripts name the alternatives too.
+
+- **`serverag` no longer fails silently.** With no embedding or completion
+  model configured — the state a first-time user is in — it exited 1 with
+  nothing on stdout or stderr, in both web and `--interactive` mode. The
+  validation message existed but went only to `logger.error`, and the library
+  attaches a `NullHandler` to its own logger, so `logging.lastResort` never
+  fired and the message was discarded unless the user happened to pass
+  `--logger_levels`. The startup failure now goes straight to stderr
+  (`Error: Failed to initialize RAG pipeline: ... Check
+  --embedding_model/--embedding_source ...`), the traceback stays on the debug
+  log, and logging is configured on every run as `makevectordatabase` and
+  `chatterlang_script` already did.
+
+- **A component that rejects its options is now a `CompileError`, not a
+  traceback.** ChatterLang compilation wrapped only `TypeError` from a
+  segment's or source's constructor, so the most common newcomer mistakes —
+  `llmPrompt[source="llamacpp"]`, or `llmPrompt` with no model configured —
+  escaped as raw `ValueError`s: `chatterlang_script` printed a full Python
+  traceback, and `--verbose` made no difference because the friendly handler
+  never saw them. Such failures now read
+  `Error: Segment 'llmPrompt' could not be created: <why>`, with the traceback
+  behind `--verbose`, and the workbench's lint endpoint reports them like any
+  other compile error.
+
+- **Unsupported LLM sources say what is supported.** `llmPrompt` and
+  `llmVisionPrompt` reported only `Unknown source: <name>`; they now match
+  `llmEmbed`: `Source 'llamacpp' is not supported. Supported sources are:
+  ['anthropic', 'eliza', 'ollama', 'openai']`. The chat and embedding segments
+  also no longer log these messages in addition to raising them, which printed
+  them twice.
+
+- **Documentation corrections**, mostly found by walking the docs as a
+  first-time reader:
+  - The README's RAG-at-a-glance intro said questions were asked "against the
+    App Center" — a find/replace slip; it is the vector store. The `tmp://`
+    scheme it uses is now explained (in-process, discarded at exit) there and
+    in the model2vec guide, where splitting the example across two processes
+    silently returned "No relevant information found."
+  - The model2vec guide named `potion-base-8M` as the default model with 256
+    dimensions; the default is `potion-retrieval-32M` with 512. The model
+    table gained a dimensions column, the "256" comment now names the model
+    the snippet actually passes, the Python-API example passes the model it
+    precaches, and the offline section warns that a model missing from the
+    cache fails under `HF_HUB_OFFLINE=1` with an error that does not name it.
+  - The reference-browser walkthrough showed invented transcripts —
+    `filterEmails`, `extractAttachments`, `mongoConnect`, `mongoQuery`,
+    `parseEmailHeaders`, and a `readEmail` parameter list the source never
+    had. Replaced with real output.
+  - `chatterlang_serve`'s reference now says that `/history` and the SSE
+    stream are scoped to the `talkpipe_session_id` cookie (its own `curl`
+    example returned an empty history), that results come back in the
+    `/process` response rather than over `/output-stream`, that `--script` is
+    optional, and that `/docs` exists.
+  - Fixed two dead anchors (the model/source guide's own table of contents,
+    and the plugin manager's link into the extension guide), removed a
+    troubleshooting row that told readers to check `DEFAULT_*` keys the same
+    page says do not exist, documented `chatterlang_script --verbose`, noted
+    that the ChatterLang architecture page's Ollama examples work with any
+    provider, added the model2vec and lazy-loading pages to the docs index,
+    showed `makevectordatabase`'s expected output, quoted the
+    `pip install "talkpipe[...]"` extras so they work in zsh, and pointed the
+    Python prerequisite at the versions CI actually tests.
+
+- `chatterlang_workbench` now exits within a couple of seconds of Ctrl-C even
+  while a request is still running. It used to wait for that request to
+  finish — a script started from the editor, or the suggestions sidebar's LLM
+  call, could hold the process for a minute or more — because uvicorn's
+  graceful shutdown had no time limit, and a second Ctrl-C did not help
+  because Python then waited for the worker thread the request ran on before
+  exiting. Running requests now get a two-second grace period, after which
+  the workbench exits without waiting for them.
+- The `pypdf` extra now requires pypdf 6.16.1 or later. Earlier releases can
+  be driven into an infinite loop by a crafted PDF through
+  `TreeObject.insert_child`; the floor is a security minimum, and every
+  fresh install already resolved above it.
+
 ## 1.0.1
 
 - The Anthropic adapter no longer sends the `temperature` parameter. The
@@ -9,7 +142,6 @@
   probes and LLM summarization — failed with
   `Messages.create() got an unexpected keyword argument 'temperature'`).
   A requested temperature is now dropped with a warning in the log instead.
-
 ## 1.0.0
 
 - `ShingleText` and `AbstractFieldSegment` no longer call the item's `.copy()`

{talkpipe-1.0.1 → talkpipe-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: talkpipe
-Version: 1.0.1
+Version: 1.0.2
 Summary: Python internal and external DSL for writing generative AI analytics
 Author-email: Travis Bauer <tlbauer@sandia.gov>
 Maintainer-email: Travis Bauer <tlbauer@sandia.gov>
@@ -56,7 +56,7 @@ Requires-Dist: openai; extra == "openai"
 Provides-Extra: anthropic
 Requires-Dist: anthropic>=0.87.0; extra == "anthropic"
 Provides-Extra: pypdf
-Requires-Dist: pypdf>=6.15.0; extra == "pypdf"
+Requires-Dist: pypdf>=6.16.1; extra == "pypdf"
 Provides-Extra: pillow
 Requires-Dist: Pillow>=12.2.0; extra == "pillow"
 Provides-Extra: model2vec
@@ -106,15 +106,12 @@ For the full story, see **[Architecture](docs/architecture/)**.
 
 ### Key Applications
 
-These are the entry points for different usage scenarios, from interactive development to production deployment:
+Entry points for different usage scenarios, from interactive development to production deployment:
 
 - **[chatterlang_workbench](docs/api-reference/chatterlang-workbench.md)**
   A browser-based IDE for writing, testing, and running ChatterLang scripts: editor with autocomplete and live error checking, real-time execution, pipeline save/load, next-component suggestions, logging, and documentation lookup.
 
-<video width="640" height="360" controls>
-  <source src="docs/workbench_demo.mp4" type="video/mp4">
-  Your browser does not support the video tag.
-</video>
+<center><img src="docs/chatterlang_workbench_screenshot.png" width=700 alt="ChatterLang Workbench: pipeline editor with autocomplete, next-component suggestions, and pipeline output"></center>
 
 - **[chatterlang_script](docs/api-reference/chatterlang-script.md)**
   Runs ChatterLang scripts from files or directly from the command line, for batch processing and automation.
@@ -134,27 +131,42 @@ These are the entry points for different usage scenarios, from interactive devel
 - **[Container images](docs/guides/container-images.md)**
   Pull release images from GitHub Container Registry (multi-platform on each GitHub release).
 
+- **Applications built on TalkPipe**
+  Install them as standalone commands with [uv](https://docs.astral.sh/uv/getting-started/installation/), each into its own environment:
+
+  | Application | Install | Run |
+  |---|---|---|
+  | [TalkPipe Vault](https://github.com/sandialabs/talkpipe-vault) — chat with your own documents | `uv tool install talkpipe-vault` | `vault-server` |
+  | [TalkPipe Writing Assistant](https://github.com/sandialabs/talkpipe-writing-assistant) — multi-user writing assistant | `uv tool install talkpipe-writing-assistant` | `writing-assistant` |
+  | ChatterLang Workbench and the other TalkPipe commands | `uv tool install "talkpipe[all]"` | `chatterlang_workbench` |
+
+  `uv tool upgrade <package>` upgrades one, `uv tool uninstall <package>` removes it, and `uv tool update-shell` puts the commands on your `PATH` if they are not found. For a pre-release, add `--prerelease allow` to the install; to go back to the latest release afterwards, reinstall with `--reinstall` (`uv tool install --upgrade` never moves a package backwards).
+
 ## Quick Start
 
-**Requirements:** Python 3.11 or newer. Check your version first with `python3 --version` — it must report 3.11 or higher before installing.
+**Requirements:** Python 3.11 or newer (3.11, 3.12, and 3.13 are the versions tested in CI). Check yours with `python3 --version` before installing.
 
 ```bash
 pip install talkpipe
 ```
 
-For LLM support, install the provider(s) you need:
+TalkPipe is provider-neutral: it works with Ollama (local or remote), OpenAI, and Anthropic for chat, and with Ollama, OpenAI, and model2vec (in-process, no server) for embeddings. None of them is required; install the provider(s) you need:
 
 ```bash
-pip install talkpipe[openai]    # OpenAI
-pip install talkpipe[ollama]    # Ollama
-pip install talkpipe[anthropic] # Anthropic Claude
-pip install talkpipe[model2vec] # In-process static embeddings (also in [all])
-# Or: pip install talkpipe[all]
+pip install "talkpipe[openai]"    # OpenAI
+pip install "talkpipe[ollama]"    # Ollama
+pip install "talkpipe[anthropic]" # Anthropic Claude
+pip install "talkpipe[model2vec]" # In-process static embeddings (also in [all])
+# Or: pip install "talkpipe[all]"
 ```
 
+(The quotes matter in zsh, where an unquoted `[...]` is a glob.)
+
+See **[LLM providers](docs/guides/model-and-source-configuration.md#llm-providers)** for what each provider needs and how to select it.
+
 > **Any provider works in any example.** The examples in this README mostly show `source="ollama"`, but that is just a per-segment parameter: swap in `source="openai"` or `source="anthropic"` (with a matching `model`) on any LLM segment — the RAG helpers take the same choice as `embedding_source`/`completion_source`. Different segments in one pipeline can even use different providers. Installing `talkpipe[all]` includes all provider integrations, so switching or mixing needs no further installs.
 
-Configure API keys and provider URLs via environment variables (for example `OPENAI_API_KEY` and `ANTHROPIC_API_KEY`) or `~/.talkpipe.toml`. If TalkPipe runs on a different machine than your Ollama server, set `TALKPIPE_OLLAMA_SERVER_URL` to that host, e.g. `export TALKPIPE_OLLAMA_SERVER_URL="http://<ollama host ip>:11434"` (a bare host/IP with no scheme or port, like `"myollamahost"`, also works). Note that the model must already be pulled **on that server** — run `ollama pull llama3.2` there, or `OLLAMA_HOST=http://<ollama host ip>:11434 ollama pull llama3.2` from your machine. See **[Configuration](docs/architecture/configuration.md)** for details and ChatterLang `$var` substitution.
+Provider API keys are read from the environment by the providers' own SDKs (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`), not from `~/.talkpipe.toml`. If TalkPipe runs on a different machine than your Ollama server, set `TALKPIPE_OLLAMA_SERVER_URL` to that host, e.g. `export TALKPIPE_OLLAMA_SERVER_URL="http://<ollama host ip>:11434"` (a bare host/IP with no scheme or port, like `"myollamahost"`, also works). The model must already be pulled **on that server** — run `ollama pull llama3.2` there, or `OLLAMA_HOST=http://<ollama host ip>:11434 ollama pull llama3.2` from your machine. See **[Configuration](docs/architecture/configuration.md)** for details and ChatterLang `$var` substitution.
 
 Hello world (no LLM server required):
 
@@ -172,9 +184,9 @@ chat("What's my name?")
 
 > **Tip:** `from talkpipe.chatterlang import compile` is equivalent to the `compiler.compile(...)` style above — both styles work throughout the docs.
 
-Multi-turn chat (requires a local model server):
+Multi-turn chat (requires an LLM provider):
 
-> **Prerequisite:** this example uses Ollama, a separate application (not just the `talkpipe[ollama]` Python package). Install and start it from https://ollama.com/download, then pull the model with `ollama pull llama3.2`. Cloud users can skip Ollama and substitute `source="openai"` or `source="anthropic"` (see the commented variants below).
+> **Prerequisite:** as written, this example uses Ollama, a separate application (not just the `talkpipe[ollama]` Python package). Install and start it from https://ollama.com/download, then pull the model with `ollama pull llama3.2`. Ollama is not required: to use OpenAI or Anthropic instead, substitute `source="openai"` or `source="anthropic"` (see the commented variants below).
 
 <!-- doc-example: requires-ollama -->
 ```python
@@ -191,7 +203,7 @@ response = chat("What's my name?")  # Will remember context
 
 ### RAG at a glance
 
-Index a list of strings, then ask questions against the store (expand with options in [Example 5](#example-5-rag-pipeline-with-vector-database)):
+Index a list of strings, then ask questions against that vector store (expand with options in [Example 5](#example-5-rag-pipeline-with-vector-database)). `tmp://my_kb` is an in-process store that lives only as long as the interpreter — both halves below must run in the same process; use a directory path (as [Example 5](#example-5-rag-pipeline-with-vector-database) does) to keep the index:
 
 <!-- doc-example: requires-ollama -->
 ```python
@@ -209,7 +221,7 @@ rag = compiler.compile(
 rag("What is TalkPipe?")
 ```
 
-> **No Ollama server?** Swap `embedding_source="model2vec"` and `embedding_model="minishlab/potion-base-8M"` for offline embeddings (included in `talkpipe[all]`). The first run downloads the model from Hugging Face (a few files, ~30 MB); after that it's cached and needs no network — see [Precache for offline use](docs/guides/model2vec-embeddings.md#precache-for-offline-use) to pre-download for air-gapped environments. Note: the `ragToText` completion step still requires an LLM provider. See the [model2vec guide](docs/guides/model2vec-embeddings.md).
+> **No Ollama server?** Swap `embedding_source="model2vec"` and `embedding_model="minishlab/potion-base-8M"` for offline embeddings (included in `talkpipe[all]`). The first run downloads that model from Hugging Face (a few files, tens of MB); after that it's cached and needs no network — see [Precache for offline use](docs/guides/model2vec-embeddings.md#precache-for-offline-use) to pre-download for air-gapped environments. The `ragToText` completion step still needs a chat provider: set `completion_source="openai"` or `completion_source="anthropic"` with a matching `completion_model`. (OpenAI embeddings work too: `embedding_source="openai"`, `embedding_model="text-embedding-3-small"`.) See the [model2vec guide](docs/guides/model2vec-embeddings.md).
 
 > **Indexing large collections inside a container?** Building a vector
 > database over thousands of documents (`makevectordatabase`,
@@ -299,7 +311,7 @@ result = pipeline()
 # Returns: ['HELLO', 'WORLD']
 ```
 
-The `@registry.register_segment()` decorator makes your component discoverable by ChatterLang's compiler, allowing you to use it in scripts alongside built-in segments.
+The `@registry.register_segment()` decorator makes your component discoverable by ChatterLang's compiler, so scripts can use it alongside built-in segments.
 
 ### Key ChatterLang Features
 
@@ -574,7 +586,7 @@ talkpipe/
 ├── app/          # Runnable applications (servers, CLIs)
 ├── chatterlang/  # ChatterLang parser, compiler, and components
 ├── data/         # Data manipulation and I/O components
-├── llm/          # LLM integrations (OpenAI, Ollama, Anthropic)
+├── llm/          # LLM and embedding integrations (Ollama, OpenAI, Anthropic, model2vec)
 ├── operations/   # Algorithms and data processing
 ├── pipe/         # Core pipeline infrastructure
 ├── pipelines/    # High-level pipeline components (RAG, vector DB)
@@ -584,7 +596,7 @@ talkpipe/
 
 ## Configuration
 
-TalkPipe uses a flexible configuration system via `~/.talkpipe.toml` or environment variables. For LLM and embedding `model` / `source` defaults, see [Model and source configuration](docs/guides/model-and-source-configuration.md).
+TalkPipe uses a flexible configuration system via `~/.talkpipe.toml` or environment variables. For LLM and embedding `model` / `source` defaults, see [Model and source configuration](docs/guides/model-and-source-configuration.md). The example below makes Ollama the default chat provider; `default_model_source = "openai"` or `"anthropic"` (with a matching `default_model_name`) works the same way.
 
 ```toml
 # ~/.talkpipe.toml
@@ -692,6 +704,17 @@ We welcome contributions! Whether it's new components, bug fixes, documentation,
 
 ### Development environment
 
+The default branch, `stable`, is release-only: it points at the latest
+release, so what you see on the repository's front page describes that
+release. Development happens on `main`, which is where merge requests go and
+where unreleased changes and their documentation accumulate — check it out
+first:
+
+```bash
+git clone https://github.com/sandialabs/talkpipe.git
+cd talkpipe && git checkout main
+```
+
 Local development uses [uv](https://docs.astral.sh/uv/) against the committed
 `uv.lock`, so contributors share one reproducible set of versions:
 
@@ -702,8 +725,8 @@ uv sync --all-extras
 **CI does not use the lockfile.** It installs with pip (`pip install -e
 ".[dev,all]"`) and resolves dependencies fresh, on purpose: that is what
 someone running `pip install talkpipe` gets, so the build breaks when *they*
-would break. A dependency problem that only the lockfile hides is a problem
-we want CI to see.
+would break — a dependency problem the lockfile would hide is exactly what we
+want CI to see.
 
 Two consequences worth remembering:
 

{talkpipe-1.0.1 → talkpipe-1.0.2}/README.md

@@ -39,15 +39,12 @@ For the full story, see **[Architecture](docs/architecture/)**.
 
 ### Key Applications
 
-These are the entry points for different usage scenarios, from interactive development to production deployment:
+Entry points for different usage scenarios, from interactive development to production deployment:
 
 - **[chatterlang_workbench](docs/api-reference/chatterlang-workbench.md)**
   A browser-based IDE for writing, testing, and running ChatterLang scripts: editor with autocomplete and live error checking, real-time execution, pipeline save/load, next-component suggestions, logging, and documentation lookup.
 
-<video width="640" height="360" controls>
-  <source src="docs/workbench_demo.mp4" type="video/mp4">
-  Your browser does not support the video tag.
-</video>
+<center><img src="docs/chatterlang_workbench_screenshot.png" width=700 alt="ChatterLang Workbench: pipeline editor with autocomplete, next-component suggestions, and pipeline output"></center>
 
 - **[chatterlang_script](docs/api-reference/chatterlang-script.md)**
   Runs ChatterLang scripts from files or directly from the command line, for batch processing and automation.
@@ -67,27 +64,42 @@ These are the entry points for different usage scenarios, from interactive devel
 - **[Container images](docs/guides/container-images.md)**
   Pull release images from GitHub Container Registry (multi-platform on each GitHub release).
 
+- **Applications built on TalkPipe**
+  Install them as standalone commands with [uv](https://docs.astral.sh/uv/getting-started/installation/), each into its own environment:
+
+  | Application | Install | Run |
+  |---|---|---|
+  | [TalkPipe Vault](https://github.com/sandialabs/talkpipe-vault) — chat with your own documents | `uv tool install talkpipe-vault` | `vault-server` |
+  | [TalkPipe Writing Assistant](https://github.com/sandialabs/talkpipe-writing-assistant) — multi-user writing assistant | `uv tool install talkpipe-writing-assistant` | `writing-assistant` |
+  | ChatterLang Workbench and the other TalkPipe commands | `uv tool install "talkpipe[all]"` | `chatterlang_workbench` |
+
+  `uv tool upgrade <package>` upgrades one, `uv tool uninstall <package>` removes it, and `uv tool update-shell` puts the commands on your `PATH` if they are not found. For a pre-release, add `--prerelease allow` to the install; to go back to the latest release afterwards, reinstall with `--reinstall` (`uv tool install --upgrade` never moves a package backwards).
+
 ## Quick Start
 
-**Requirements:** Python 3.11 or newer. Check your version first with `python3 --version` — it must report 3.11 or higher before installing.
+**Requirements:** Python 3.11 or newer (3.11, 3.12, and 3.13 are the versions tested in CI). Check yours with `python3 --version` before installing.
 
 ```bash
 pip install talkpipe
 ```
 
-For LLM support, install the provider(s) you need:
+TalkPipe is provider-neutral: it works with Ollama (local or remote), OpenAI, and Anthropic for chat, and with Ollama, OpenAI, and model2vec (in-process, no server) for embeddings. None of them is required; install the provider(s) you need:
 
 ```bash
-pip install talkpipe[openai]    # OpenAI
-pip install talkpipe[ollama]    # Ollama
-pip install talkpipe[anthropic] # Anthropic Claude
-pip install talkpipe[model2vec] # In-process static embeddings (also in [all])
-# Or: pip install talkpipe[all]
+pip install "talkpipe[openai]"    # OpenAI
+pip install "talkpipe[ollama]"    # Ollama
+pip install "talkpipe[anthropic]" # Anthropic Claude
+pip install "talkpipe[model2vec]" # In-process static embeddings (also in [all])
+# Or: pip install "talkpipe[all]"
 ```
 
+(The quotes matter in zsh, where an unquoted `[...]` is a glob.)
+
+See **[LLM providers](docs/guides/model-and-source-configuration.md#llm-providers)** for what each provider needs and how to select it.
+
 > **Any provider works in any example.** The examples in this README mostly show `source="ollama"`, but that is just a per-segment parameter: swap in `source="openai"` or `source="anthropic"` (with a matching `model`) on any LLM segment — the RAG helpers take the same choice as `embedding_source`/`completion_source`. Different segments in one pipeline can even use different providers. Installing `talkpipe[all]` includes all provider integrations, so switching or mixing needs no further installs.
 
-Configure API keys and provider URLs via environment variables (for example `OPENAI_API_KEY` and `ANTHROPIC_API_KEY`) or `~/.talkpipe.toml`. If TalkPipe runs on a different machine than your Ollama server, set `TALKPIPE_OLLAMA_SERVER_URL` to that host, e.g. `export TALKPIPE_OLLAMA_SERVER_URL="http://<ollama host ip>:11434"` (a bare host/IP with no scheme or port, like `"myollamahost"`, also works). Note that the model must already be pulled **on that server** — run `ollama pull llama3.2` there, or `OLLAMA_HOST=http://<ollama host ip>:11434 ollama pull llama3.2` from your machine. See **[Configuration](docs/architecture/configuration.md)** for details and ChatterLang `$var` substitution.
+Provider API keys are read from the environment by the providers' own SDKs (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`), not from `~/.talkpipe.toml`. If TalkPipe runs on a different machine than your Ollama server, set `TALKPIPE_OLLAMA_SERVER_URL` to that host, e.g. `export TALKPIPE_OLLAMA_SERVER_URL="http://<ollama host ip>:11434"` (a bare host/IP with no scheme or port, like `"myollamahost"`, also works). The model must already be pulled **on that server** — run `ollama pull llama3.2` there, or `OLLAMA_HOST=http://<ollama host ip>:11434 ollama pull llama3.2` from your machine. See **[Configuration](docs/architecture/configuration.md)** for details and ChatterLang `$var` substitution.
 
 Hello world (no LLM server required):
 
@@ -105,9 +117,9 @@ chat("What's my name?")
 
 > **Tip:** `from talkpipe.chatterlang import compile` is equivalent to the `compiler.compile(...)` style above — both styles work throughout the docs.
 
-Multi-turn chat (requires a local model server):
+Multi-turn chat (requires an LLM provider):
 
-> **Prerequisite:** this example uses Ollama, a separate application (not just the `talkpipe[ollama]` Python package). Install and start it from https://ollama.com/download, then pull the model with `ollama pull llama3.2`. Cloud users can skip Ollama and substitute `source="openai"` or `source="anthropic"` (see the commented variants below).
+> **Prerequisite:** as written, this example uses Ollama, a separate application (not just the `talkpipe[ollama]` Python package). Install and start it from https://ollama.com/download, then pull the model with `ollama pull llama3.2`. Ollama is not required: to use OpenAI or Anthropic instead, substitute `source="openai"` or `source="anthropic"` (see the commented variants below).
 
 <!-- doc-example: requires-ollama -->
 ```python
@@ -124,7 +136,7 @@ response = chat("What's my name?")  # Will remember context
 
 ### RAG at a glance
 
-Index a list of strings, then ask questions against the store (expand with options in [Example 5](#example-5-rag-pipeline-with-vector-database)):
+Index a list of strings, then ask questions against that vector store (expand with options in [Example 5](#example-5-rag-pipeline-with-vector-database)). `tmp://my_kb` is an in-process store that lives only as long as the interpreter — both halves below must run in the same process; use a directory path (as [Example 5](#example-5-rag-pipeline-with-vector-database) does) to keep the index:
 
 <!-- doc-example: requires-ollama -->
 ```python
@@ -142,7 +154,7 @@ rag = compiler.compile(
 rag("What is TalkPipe?")
 ```
 
-> **No Ollama server?** Swap `embedding_source="model2vec"` and `embedding_model="minishlab/potion-base-8M"` for offline embeddings (included in `talkpipe[all]`). The first run downloads the model from Hugging Face (a few files, ~30 MB); after that it's cached and needs no network — see [Precache for offline use](docs/guides/model2vec-embeddings.md#precache-for-offline-use) to pre-download for air-gapped environments. Note: the `ragToText` completion step still requires an LLM provider. See the [model2vec guide](docs/guides/model2vec-embeddings.md).
+> **No Ollama server?** Swap `embedding_source="model2vec"` and `embedding_model="minishlab/potion-base-8M"` for offline embeddings (included in `talkpipe[all]`). The first run downloads that model from Hugging Face (a few files, tens of MB); after that it's cached and needs no network — see [Precache for offline use](docs/guides/model2vec-embeddings.md#precache-for-offline-use) to pre-download for air-gapped environments. The `ragToText` completion step still needs a chat provider: set `completion_source="openai"` or `completion_source="anthropic"` with a matching `completion_model`. (OpenAI embeddings work too: `embedding_source="openai"`, `embedding_model="text-embedding-3-small"`.) See the [model2vec guide](docs/guides/model2vec-embeddings.md).
 
 > **Indexing large collections inside a container?** Building a vector
 > database over thousands of documents (`makevectordatabase`,
@@ -232,7 +244,7 @@ result = pipeline()
 # Returns: ['HELLO', 'WORLD']
 ```
 
-The `@registry.register_segment()` decorator makes your component discoverable by ChatterLang's compiler, allowing you to use it in scripts alongside built-in segments.
+The `@registry.register_segment()` decorator makes your component discoverable by ChatterLang's compiler, so scripts can use it alongside built-in segments.
 
 ### Key ChatterLang Features
 
@@ -507,7 +519,7 @@ talkpipe/
 ├── app/          # Runnable applications (servers, CLIs)
 ├── chatterlang/  # ChatterLang parser, compiler, and components
 ├── data/         # Data manipulation and I/O components
-├── llm/          # LLM integrations (OpenAI, Ollama, Anthropic)
+├── llm/          # LLM and embedding integrations (Ollama, OpenAI, Anthropic, model2vec)
 ├── operations/   # Algorithms and data processing
 ├── pipe/         # Core pipeline infrastructure
 ├── pipelines/    # High-level pipeline components (RAG, vector DB)
@@ -517,7 +529,7 @@ talkpipe/
 
 ## Configuration
 
-TalkPipe uses a flexible configuration system via `~/.talkpipe.toml` or environment variables. For LLM and embedding `model` / `source` defaults, see [Model and source configuration](docs/guides/model-and-source-configuration.md).
+TalkPipe uses a flexible configuration system via `~/.talkpipe.toml` or environment variables. For LLM and embedding `model` / `source` defaults, see [Model and source configuration](docs/guides/model-and-source-configuration.md). The example below makes Ollama the default chat provider; `default_model_source = "openai"` or `"anthropic"` (with a matching `default_model_name`) works the same way.
 
 ```toml
 # ~/.talkpipe.toml
@@ -625,6 +637,17 @@ We welcome contributions! Whether it's new components, bug fixes, documentation,
 
 ### Development environment
 
+The default branch, `stable`, is release-only: it points at the latest
+release, so what you see on the repository's front page describes that
+release. Development happens on `main`, which is where merge requests go and
+where unreleased changes and their documentation accumulate — check it out
+first:
+
+```bash
+git clone https://github.com/sandialabs/talkpipe.git
+cd talkpipe && git checkout main
+```
+
 Local development uses [uv](https://docs.astral.sh/uv/) against the committed
 `uv.lock`, so contributors share one reproducible set of versions:
 
@@ -635,8 +658,8 @@ uv sync --all-extras
 **CI does not use the lockfile.** It installs with pip (`pip install -e
 ".[dev,all]"`) and resolves dependencies fresh, on purpose: that is what
 someone running `pip install talkpipe` gets, so the build breaks when *they*
-would break. A dependency problem that only the lockfile hides is a problem
-we want CI to see.
+would break — a dependency problem the lockfile would hide is exactly what we
+want CI to see.
 
 Two consequences worth remembering:
 

{talkpipe-1.0.1 → talkpipe-1.0.2}/RELEASING.md

@@ -34,13 +34,22 @@ history and only made version sorting harder. Semantic versioning rules
    git push origin vX.Y.Z
    ```
 
+   Then advance `stable`, the repository's default branch, to the release.
+   `stable` only ever points at a full release, so skip this for betas and
+   release candidates:
+
+   ```bash
+   git push origin vX.Y.Z^{commit}:stable
+   ```
+
 4. **Publish a release** for the tag (Gitea: *Releases → New Release*;
    GitHub mirror: *Releases → Draft a new release*). The CI workflow's
    `release: published` trigger runs `publish-package`, which builds the
    sdist and wheel, `twine check`s them, and uploads to PyPI with the
    `PYPI_API_TOKEN` repository secret. Mark betas and release candidates as
    pre-releases. Watch the run finish; the package job and the container build
-   (GitHub only) run in the same workflow.
+   (GitHub only) run in the same workflow. `publish-package` ignores any tag
+   that does not start with `v`, so a stray publish cannot upload to PyPI.
 5. **Verify** with a fresh environment: `pip install talkpipe==X.Y.Z`,
    `python -c "import talkpipe; print(talkpipe.__version__)"`.
 
@@ -50,6 +59,6 @@ Nothing is bumped afterwards; the next commit on `main` reports itself as
 ## Downstream
 
 The consumers in this suite pin a floor on talkpipe (`talkpipe_vault`,
-`talkpipe-agents`, `talkpipe-writing-assistant`, `privateer`, `local-m2v`).
+`talkpipe-agents`, `talkpipe-writing-assistant`, `local-m2v`).
 When a release adds API they rely on, or when the major version changes,
 raise their floors in a follow-up merge request in each repo.

{talkpipe-1.0.1 → talkpipe-1.0.2}/docs/README.md

@@ -11,7 +11,10 @@ New to TalkPipe? Start here for installation, basic concepts, and your first pip
 Create vector databases and run RAG servers in two commands: `makevectordatabase` and `serverag`.
 
 ### ⚙️ [Model and source configuration](guides/model-and-source-configuration.md)
-Set default LLM and embedding providers via segment parameters, `~/.talkpipe.toml`, or environment variables.
+The supported LLM providers — Ollama, OpenAI, and Anthropic for chat; Ollama, OpenAI, and in-process model2vec for embeddings — what each needs, and how to select one per segment, in `~/.talkpipe.toml`, or with environment variables.
+
+### 🧮 [Model2vec embeddings](guides/model2vec-embeddings.md)
+In-process static embeddings: no server, no API key, and no network once the model is cached — including how to precache for air-gapped environments.
 
 ### 🐳 [Container images](guides/container-images.md)
 Pull multi-platform release images from GitHub Container Registry (`ghcr.io`), tags, and Docker/Podman usage.
@@ -30,6 +33,7 @@ Complete documentation for all TalkPipe commands and components:
 - [ChatterLang Script Runner](api-reference/chatterlang-script.md) - Run a script from the command line
 - [Documentation Generator](api-reference/talkpipe-ref.md) - Generate reference documentation for Segments and Sources
 - [Plugin Manager](api-reference/talkpipe-plugin-manager.md) - Manage and inspect TalkPipe plugins
+- [Lazy loading](api-reference/lazy-loading.md) - How the registry imports a component's module only when it is named
 
 ### 🏗️ [Architecture](architecture/)
 Deep technical documentation:

{talkpipe-1.0.1 → talkpipe-1.0.2}/docs/api-reference/chatterlang-script.md

@@ -4,7 +4,7 @@
 
 ## Overview
 
-ChatterLang Script is a command-line interface for running ChatterLang pipeline scripts. It compiles and executes scripts written in TalkPipe's external DSL. This tool is useful for automation, batch data processing, and integration into shell scripts or CI/CD pipelines. It supports inline scripts, file-based scripts, configuration-based scripts, and custom module loading.
+`chatterlang_script` compiles and executes scripts written in TalkPipe's external DSL, which makes it useful for automation, batch data processing, and integration into shell scripts or CI/CD pipelines. A script can come from a file, a configuration key, or inline text, and custom modules can be loaded alongside it.
 
 ## Usage
 
@@ -22,6 +22,7 @@ chatterlang_script --script "script_content" [options]
 | `--load-module` | string | Path to custom module file to import before execution (can be specified multiple times) |
 | `--logger_levels` | string | Logger level configuration in format 'logger:level,logger:level,...' |
 | `--logger_files` | string | Logger file output configuration in format 'logger:file,logger:file,...' |
+| `--verbose` | flag | Show the full Python traceback on compile and runtime errors instead of just the one-line `Error: ...` message |
 | `--<key>` | any | Any additional argument becomes a configuration value accessible via `$key` syntax in the script |
 
 ## Script Sources
@@ -86,6 +87,8 @@ This feature is useful for parameterizing scripts without editing configuration
 - Check input data format matches expected types
 - Verify external service connectivity (APIs, URLs)
 - Review ChatterLang syntax for typos
+- Errors are reported as a single `Error: ...` line; add `--verbose` for the full
+  Python traceback
 
 **Performance issues:**
 - Monitor memory usage with large datasets
@@ -94,7 +97,7 @@ This feature is useful for parameterizing scripts without editing configuration
 
 ### Debug Mode
 
-Enable comprehensive logging:
+Enable detailed logging:
 
 ```bash
 chatterlang_script \