TalkPipe 0.9.2__tar.gz → 0.9.4__tar.gz

{talkpipe-0.9.2 → talkpipe-0.9.4}/.github/workflows/ci-cd.yml

@@ -148,7 +148,7 @@ jobs:
         output: 'trivy-results.sarif'
     
     - name: Upload Trivy scan results to GitHub Security tab
-      uses: github/codeql-action/upload-sarif@v3
+      uses: github/codeql-action/upload-sarif@v4
       if: always()
       with:
         sarif_file: 'trivy-results.sarif'
@@ -171,15 +171,15 @@ jobs:
       uses: actions/checkout@v5
     
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v3
+      uses: github/codeql-action/init@v4
       with:
         languages: ${{ matrix.language }}
     
     - name: Autobuild
-      uses: github/codeql-action/autobuild@v3
+      uses: github/codeql-action/autobuild@v4
     
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v3
+      uses: github/codeql-action/analyze@v4
 
   publish-package:
     runs-on: ubuntu-latest

{talkpipe-0.9.2 → talkpipe-0.9.4}/CHANGELOG.md

@@ -1,5 +1,52 @@
 # Changelog
 
+## 0.9.4
+### Improvements
+- Changes several logging statements to make the consistent with the rest of the code
+- Refactored **lambda** to use AbstractFieldSegment for consistency
+- Removed fail_on_error parameter in **lambda** and **lambdaFilter**.  This is an API breaking change.
+  The segments will now always fail on an error and there is no option to silently fail.
+- Changed chatterlang_serve so that it raises and exception an exits if given a script that can't be
+  compiled.  The previous behavior was that it would issue a log message and fall back to the default
+  script.
+- Added unit test for parse_unknown_args and expanded it to support boolean flag parameters
+- Marked UMAP as deprecated.  It will be removed in 1.0.  It requires additional dependencies that no
+  other core modules need.
+- Updated the ollama embedding and chat connector so that it uses the OLLAMA_SERVER_URL configuration
+  variable as the host where ollama is installed.  So OLLAMA_SERVER_URL can be set in the TOML configuration
+  file or TALKPIPE_OLLAMA_SERVER_URL can be set as an environment variable.
+- Implemented better compile errors for when sources or segments are not found.  It had been a key error.
+  It will now be a compile error.
+- Added batch files for the tutorials so it is easier to run them on Windows
+- Enhanced **prompt** source to catch downstream pipeline errors and continue prompting instead of
+  crashing. When a user enters invalid input that causes an exception in downstream segments, the error
+  is displayed with a full traceback, and the prompt continues accepting input. This makes interactive
+  pipelines more robust and user-friendly. The error-resilient behavior is enabled by default but can
+  be disabled by setting `error_resilient=False`.
+- Updated **chatterlang_serve** to support API key configuration via the configuration system. When
+  `--api-key` is not specified on the command line, it will check for `API_KEY` in the configuration,
+  which automatically checks the `TALKPIPE_API_KEY` environment variable. This makes it easier to
+  deploy chatterlang_serve in Docker and CI/CD environments without exposing API keys in command history.
+- Added support for single-quoted strings in ChatterLang scripts. Single quotes work identically to
+  double quotes, including support for escaping quotes by doubling them (`''` → `'`). This provides
+  more flexibility when writing scripts, especially when the string content contains double quotes.
+- Enhanced **echo** source with an `n` parameter to repeat the output data multiple times. For example,
+  `echo[data='a,b', n=3]` will emit: a, b, a, b, a, b. This is useful for testing pipelines and
+  generating repeated test data.
+- Improved **chatterlang_serve** startup output to clearly distinguish between the user interface URL
+  and the API endpoint URL. The server now displays a formatted message showing both URLs with clear
+  labels when it starts.
+- Updated tutorial 1, step 1 to show more feedback to the user so it doesn't just seem to be hanging
+
+## 0.9.3 
+### Improvements
+- Updated tutorials to use lancedb.  
+- Moved tutorial scripts out of shell scripts into their own files
+- Fixed issue with Docker container not including all optional installation components
+- Fixed string encoding issue with web page downloads
+- Updated documentation for plugins to fix error
+- Removed invalid script in pyproject.toml
+
 ## 0.9.2
 ### Improvements
 - Added ability for sources and segments to have multiple names in chatterlang.

{talkpipe-0.9.2 → talkpipe-0.9.4}/CLAUDE.md

@@ -16,6 +16,8 @@ ChatterLang is an external DSL and is defined under talkpipe/chatterlang
 Other packages in the project other than util include implementations of pipe sources and segments
 Util contains various generally usable utility functions, expecially for basic data manipulation and configuration.
 talkpipe/README.md contains a longer overview
+After making a change, update CHANGELOG.md
+When making a change, write a unit test that fails without the change, verify that the unit test fails, make the change and then verity that the unit test now passes.  Notify the user if it is not reasonable to write the unit test.
 
 ## Important Commands
 python -m build for creating a new release

{talkpipe-0.9.2 → talkpipe-0.9.4}/Dockerfile

@@ -34,8 +34,8 @@ COPY --chown=builder:builder src/ src/
 COPY --chown=builder:builder tests/ tests/
 
 # Install Python dependencies and build the package
-RUN python3 -m pip install --user --upgrade pip setuptools wheel build
-RUN python3 -m pip install --user numpy pandas matplotlib scikit-learn scipy
+RUN python3 -m pip install --upgrade pip setuptools wheel build
+RUN python3 -m pip install numpy pandas matplotlib scikit-learn scipy
 ENV SETUPTOOLS_SCM_PRETEND_VERSION_FOR_TALKPIPE=0.1.0
 RUN python3 -m pip install --user -e .[dev,all]
 RUN python3 -m pytest --log-cli-level=DEBUG
@@ -70,7 +70,8 @@ COPY --from=builder --chown=app:app /build/dist/*.whl /tmp/
 RUN python3 -m pip install --no-cache-dir --upgrade pip && \
     python3 -m pip install --no-cache-dir \
         numpy pandas matplotlib scikit-learn scipy && \
-    python3 -m pip install --no-cache-dir /tmp/*.whl && \
+    WHEEL_FILE=$(ls /tmp/*.whl) && \
+    python3 -m pip install --no-cache-dir "${WHEEL_FILE}[all]" && \
     rm -f /tmp/*.whl
 
 # Copy only necessary runtime files

{talkpipe-0.9.2 → talkpipe-0.9.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: talkpipe
-Version: 0.9.2
+Version: 0.9.4
 Summary: Python internal and external DSL for writing generative AI analytics
 Author-email: Travis Bauer <tlbauer@sandia.gov>
 Maintainer-email: Travis Bauer <tlbauer@sandia.gov>

{talkpipe-0.9.2 → talkpipe-0.9.4}/docs/api-reference/talkpipe-plugin-manager.md

@@ -4,7 +4,7 @@ Command-line tool for managing and inspecting TalkPipe plugins.
 
 ## Overview
 
-The `talkpipe_plugin_manager` command provides utilities to list, reload, and debug TalkPipe plugins that are installed via Python entry points.
+The `talkpipe_plugins` command provides utilities to list, reload, and debug TalkPipe plugins that are installed via Python entry points.
 
 ## Installation
 
@@ -21,7 +21,7 @@ pip install talkpipe
 Display all discovered plugins and their status:
 
 ```bash
-talkpipe_plugin_manager --list
+talkpipe_plugins --list
 ```
 
 **Example output:**
@@ -40,7 +40,7 @@ Failed plugins (1):
 Reload a specific plugin by name (useful during development):
 
 ```bash
-talkpipe_plugin_manager --reload my-data-plugin
+talkpipe_plugins --reload my-data-plugin
 ```
 
 ### Verbose Output
@@ -48,7 +48,7 @@ talkpipe_plugin_manager --reload my-data-plugin
 Show detailed information:
 
 ```bash
-talkpipe_plugin_manager --list --verbose
+talkpipe_plugins --list --verbose
 ```
 
 ## Plugin Discovery
@@ -65,7 +65,7 @@ setup(
     name="my-talkpipe-plugin",
     entry_points={
         "talkpipe.plugins": [
-            "my_plugin = my_plugin.plugin:NetworkPlugin",
+            "my_plugin = my_plugin.plugin",
         ]
     }
 )
@@ -74,21 +74,31 @@ setup(
 **pyproject.toml:**
 ```toml
 [project.entry-points."talkpipe.plugins"]
-my_plugin = "my_plugin.plugin:NetworkPlugin"
+my_plugin = "my_plugin.plugin"
 ```
 
+The entry point should reference the **module** that contains your plugin code. Component registration happens automatically through decorators when the module is imported.
+
 ## How Plugin Loading Works
 
 When TalkPipe discovers a plugin, it performs the following steps:
 
-1. **Import the module**: The plugin loader imports the module specified by the entry point (e.g., `my_plugin.plugin`)
+1. **Load the entry point**: The plugin loader loads whatever the entry point references (typically a module or function)
 2. **Automatic registration**: During import, any sources and segments defined or imported by the module are automatically registered with TalkPipe's registry system through decorators like `@registry.register_segment()`
-3. **Optional initialization**: If the loaded class/module has an `initialize_plugin` function or method, it will be called for additional setup
+3. **Optional initialization**: If the loaded object has an `initialize_plugin` function attribute, it will be called for additional setup
 
 The key insight is that **simply importing the plugin module triggers component registration**. This happens through TalkPipe's decorator-based registry system.
 
+**Important**: The entry point should reference either:
+- A **module** (e.g., `my_plugin.plugin`) that can have a module-level `initialize_plugin()` function
+- A **function** directly (e.g., `my_plugin.plugin:setup_plugin`)
+
+Entry points pointing to classes will have the `initialize_plugin` called on the **class itself** (not an instance), so it would need to be a static method or classmethod if using a class.
+
 ### Example Plugin Structure
 
+**Recommended: Module-level pattern**
+
 ```python
 # my_plugin/plugin.py
 from talkpipe.chatterlang import registry
@@ -104,19 +114,22 @@ class JsonParseSegment(AbstractSegment):
     """JSON parsing segment - registered automatically on import."""
     pass
 
-class NetworkPlugin:
-    """Main plugin class referenced by entry point."""
-    
-    def initialize_plugin(self):
-        """Optional: Called after module import for additional setup."""
-        print("Network plugin initialized!")
-        # Additional setup if needed
+def initialize_plugin():
+    """Optional: Called after module import for additional setup."""
+    print("Network plugin initialized!")
+    # Additional setup if needed
+```
+
+**Entry point configuration** (points to the module):
+```toml
+[project.entry-points."talkpipe.plugins"]
+network_plugin = "my_plugin.plugin"
 ```
 
-When this module is imported:
+When this module is loaded:
 1. The `@registry.register_source("httpGet")` decorator runs, registering the `HttpGetSource`
-2. The `@registry.register_segment("jsonParse")` decorator runs, registering the `JsonParseSegment`  
-3. The `NetworkPlugin.initialize_plugin()` method is called for any additional setup
+2. The `@registry.register_segment("jsonParse")` decorator runs, registering the `JsonParseSegment`
+3. The module-level `initialize_plugin()` function is called for any additional setup
 
 ### Alternative: Module-Level Registration
 
@@ -157,10 +170,10 @@ During plugin development, you can reload plugins without restarting Python:
 ```bash
 # Make changes to your plugin code
 # Then reload it
-talkpipe_plugin_manager --reload my-plugin
+talkpipe_plugins --reload my-plugin
 
 # Verify it loaded successfully
-talkpipe_plugin_manager --list
+talkpipe_plugins --list
 ```
 
 ## Integration with TalkPipe

{talkpipe-0.9.2 → talkpipe-0.9.4}/docs/tutorials/Tutorial_1-Document_Indexing/README.md

@@ -57,20 +57,22 @@ However, for testing and development purposes, we often need synthetic data that
 
 ### The Solution: AI-Generated Stories
 
-The first step uses TalkPipe's ChatterLang scripting language to generate 50 fictional stories about technology development. Here's what happens:
+The first step uses TalkPipe's ChatterLang scripting language to generate 50 fictional stories about technology development. The pipeline is defined in `Step_1_CreateSyntheticData.script`:
 
-```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    LOOP 50 TIMES {
-        INPUT FROM "Write a fictitious five sentence story about technology development in an imaginary country." 
-        | llmPrompt[source="ollama", model="llama3.2", multi_turn=False] 
-        | toDict[field_list="_:content"] 
-        | llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False] 
-        | dumpsJsonl | print;
-    }
-'
+```
+LOOP 50 TIMES {
+    INPUT FROM "Write a fictitious five sentence story about technology development in an imaginary country."
+    | llmPrompt[source="ollama", model="llama3.2", multi_turn=False]
+    | toDict[field_list="_:content"]
+    | llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False]
+    | dumpsJsonl | print;
+}
+```
+
+To run this script:
 
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT > stories.json
+```bash
+chatterlang_script --script Step_1_CreateSyntheticData.script > stories.json
 ```
 
 ### Breaking Down the Pipeline
@@ -166,17 +168,19 @@ Think of an index like the index at the back of a book, but much more sophistica
 
 ### The Solution: Whoosh Indexing
 
-Step 2 takes our generated stories and creates a searchable index using the Whoosh library:
+Step 2 takes our generated stories and creates a searchable index using the Whoosh library. The pipeline is defined in `Step_2_IndexStories.script`:
 
-```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    INPUT FROM "stories.json" 
-    | readJsonl 
-    | progressTicks[tick_count=1, print_count=True]
-    | indexWhoosh[index_path="./full_text_index", field_list="content,title", overwrite=True]
-'
+```
+INPUT FROM "stories.json"
+| readJsonl
+| progressTicks[tick_count=1, print_count=True]
+| indexWhoosh[index_path="./full_text_index", field_list="content,title", overwrite=True]
+```
+
+To run this script:
 
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT
+```bash
+chatterlang_script --script Step_2_IndexStories.script
 ```
 
 ### Understanding the Indexing Pipeline
@@ -283,15 +287,17 @@ Most search implementations require significant custom development, but TalkPipe
 
 ### The Solution: Dual Interface Search
 
-Step 3 creates both an API endpoint and a web interface using a single command:
+Step 3 creates both an API endpoint and a web interface using a single command. The pipeline is defined in `Step_3_SearchStories.script`:
 
-```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-  | searchWhoosh[index_path="full_text_index", field="query"] 
-  | formatItem[field_list="document.title:Title,document.content:Content,score:Score"]
-'
+```
+| searchWhoosh[index_path="full_text_index", field="query"]
+| formatItem[field_list="document.title:Title,document.content:Content,score:Score"]
+```
 
-python -m talkpipe.app.chatterlang_serve --form-config story_search_ui.yml --title "Story Search" --display-property query --script CHATTERLANG_SCRIPT
+To run this script:
+
+```bash
+chatterlang_serve --form-config story_search_ui.yml --title "Story Search" --display-property query --script Step_3_SearchStories.script
 ```
 
 ### Understanding the Search System

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_script --script Step_1_CreateSyntheticData.script

talkpipe-0.9.2/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.sh → talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.script

@@ -17,14 +17,13 @@
 # 6. The output is redirected to a file named "stories.json".
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    LOOP 50 TIMES {
-        INPUT FROM "Write a fictitious five sentence story about technology development in an imaginary country." 
-        | llmPrompt[source="ollama", model="llama3.2", multi_turn=False] 
-        | toDict[field_list="_:content"] 
-        | llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False] 
-        | dumpsJsonl | print;
-    }
-'
-#chatterlang_script --script "
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT > stories.json
+
+INPUT FROM "Generating 50 synthetic stories into stories.json" | print;
+INPUT FROM echo[data="Write a fictitious five sentence story about technology development in an imaginary country.", n=50]
+| llmPrompt[source="ollama", model="llama3.2", multi_turn=False]
+| toDict[field_list="_:content"]
+| llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False]
+| progressTicks[tick_count=1, eol_count=10, print_count=true]
+| dumpsJsonl 
+| writeString[fname="stories.json"];
+

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_script --script Step_1_CreateSyntheticData.script 

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_2_IndexStories.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_script --script Step_2_IndexStories.script

talkpipe-0.9.2/docs/tutorials/Tutorial_1-Document_Indexing/Step_2_IndexStories.sh → talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_2_IndexStories.script

@@ -6,8 +6,8 @@
 # The indexed data can then be used for full-text search.
 #
 # As a side note, the first half of this script issues a single piece of data, the
-# filename.  The next segment, `readJsonl`, reads the JSONL file line by line and 
-# issues one decoded JSON object at a time.  This is a good example of how the 
+# filename.  The next segment, `readJsonl`, reads the JSONL file line by line and
+# issues one decoded JSON object at a time.  This is a good example of how the
 # constitution of the data being processed can change as it flows through the pipeline.
 #
 # The pipeline is:
@@ -16,12 +16,7 @@
 # 3. The index is stored in the specified path "./full_text_index".
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    INPUT FROM "stories.json" 
-    | readJsonl 
-    | progressTicks[tick_count=1, print_count=True]
-    | indexWhoosh[index_path="./full_text_index", field_list="content,title", overwrite=True]
-'
-
-#chatterlang_script --script "
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT
+INPUT FROM "stories.json"
+| readJsonl
+| progressTicks[tick_count=1, print_count=True]
+| indexWhoosh[index_path="./full_text_index", field_list="content,title", overwrite=True]

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_2_IndexStories.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_script --script Step_2_IndexStories.script

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_3_SearchStories.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_serve --form-config story_search_ui.yml --title "Story Search" --display-property query --script Step_3_SearchStories.script

talkpipe-0.9.2/docs/tutorials/Tutorial_1-Document_Indexing/Step_3_SearchStories.sh → talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_3_SearchStories.script

@@ -12,8 +12,5 @@
 # needing to write any code.
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='
-  | searchWhoosh[index_path="full_text_index", field="query"] 
-  | formatItem[field_list="document.title:Title,document.content:Content,score:Score"]
-'
-chatterlang_serve --form-config story_search_ui.yml --title \"Story\ Search\" --display-property query --script CHATTERLANG_SCRIPT
+| searchWhoosh[index_path="full_text_index", field="query"]
+| formatItem[field_list="document.title:Title,document.content:Content,score:Score"]

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_3_SearchStories.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_serve --form-config story_search_ui.yml --title "Story Search" --display-property query --script Step_3_SearchStories.script

{talkpipe-0.9.2 → talkpipe-0.9.4}/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/README.md

@@ -34,16 +34,20 @@ Vector embeddings solve this by converting text into high-dimensional mathematic
 
 ### The Implementation
 
+The pipeline is defined in `Step_1_CreateVectorDatabase.script`:
+
+```
+INPUT FROM "../Tutorial_1-Document_Indexing/stories.json"
+| readJsonl
+| progressTicks[tick_count=1, print_count=True]
+| llmEmbed[field="content", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| addToLanceDB[path="./vector_index", table_name="stories", vector_field="vector", metadata_field_list="title,content", overwrite=True]
+```
+
+To run this script:
+
 ```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    INPUT FROM "../Tutorial_1-Document_Indexing/stories.json"
-    | readJsonl 
-    | progressTicks[tick_count=1, print_count=True] 
-    | llmEmbed[field="content", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | addVector[path="./vector_index", vector_field="vector", metadata_field_list="title,content", overwrite=True]
-'
-
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT
+chatterlang_script --script Step_1_CreateVectorDatabase.script
 ```
 
 ### Breaking Down the Pipeline
@@ -67,12 +71,12 @@ The `mxbai-embed-large` model is specifically designed for semantic search - it'
 
 **3. Building the Index**
 ```
-| addVector[path="./vector_index", vector_field="vector", metadata_field_list="title,content", overwrite=True]
+| addToLanceDB[path="./vector_index", table_name="stories", vector_field="vector", metadata_field_list="title,content", overwrite=True]
 ```
 This creates a specialized index that:
-- Stores vectors for similarity search
+- Stores vectors for similarity search in a LanceDB table named "stories"
 - Preserves original metadata (title and content) for retrieval
-- Enables fast nearest-neighbor queries
+- Enables fast nearest-neighbor queries using LanceDB's efficient vector search capabilities
 
 ### Real-World Applications
 
@@ -96,15 +100,19 @@ Your users don't always know the right keywords. Sometimes they have an example
 
 ### The Solution: Semantic Search Interface
 
+The pipeline is defined in `Step_2_SearchByExample.script`:
+
+```
+| copy
+| llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| searchLanceDB[field="vector", path="./vector_index", table_name="stories", limit=10]
+| formatItem[field_list="document.title:Title, document.content:Content, score:Score"]
+```
+
+To run this script:
+
 ```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='    
-    | copy
-    | llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | searchVector[vector_field="vector", path="./vector_index"]
-    | formatItem[field_list="document.title:Title, document.content:Content, score:Score"]
-'
-
-python -m talkpipe.app.chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script CHATTERLANG_SCRIPT
+chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script Step_2_SearchByExample.script
 ```
 
 ### Understanding the Search Pipeline
@@ -123,9 +131,9 @@ The user's example text is converted to a vector using the same model that index
 
 **3. Vector Search**
 ```
-| searchVector[vector_field="vector", path="./vector_index"]
+| searchLanceDB[field="vector", path="./vector_index", table_name="stories", limit=10]
 ```
-This finds the documents whose vectors are closest to the query vector - literally the nearest neighbors in high-dimensional space.
+This finds the documents whose vectors are closest to the query vector - literally the nearest neighbors in high-dimensional space. LanceDB provides efficient approximate nearest neighbor search for fast retrieval.
 
 **4. Result Formatting**
 ```
@@ -167,23 +175,27 @@ Finding relevant documents is helpful, but what users often really want is an an
 
 ### The RAG Implementation
 
+The pipeline is defined in `Step_3_SpecializedRag.script`:
+
+```
+| copy
+| llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| searchLanceDB[field="vector", path="./vector_index", table_name="stories", all_results_at_once=True, set_as="results"]
+| ragPrompt
+| llmPrompt[source="ollama", model="llama3.2"]
+```
+
+To run this script:
+
 ```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    | copy
-    | llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | searchVector[vector_field="vector", path="./vector_index", all_results_at_once=True, set_as="results"]
-    | ragPrompt
-    | llmPrompt[source="ollama", model="llama3.2"]
-'
-
-python -m talkpipe.app.chatterlang_serve --form-config story_by_example_ui.yml --load-module step_3_extras.py --display-property example --script CHATTERLANG_SCRIPT
+chatterlang_serve --form-config story_by_example_ui.yml --load-module step_3_extras.py --display-property example --script Step_3_SpecializedRag.script
 ```
 
 ### What's Different in the RAG Pipeline
 
 **1. Batch Results Collection**
 ```
-| searchVector[..., all_results_at_once=True, set_as="results"]
+| searchLanceDB[..., all_results_at_once=True, set_as="results"]
 ```
 Instead of processing results one by one, we collect all search results together. This allows the next step to see the full context.
 

talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_1_CreateVectorDatabase.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_script --script Step_1_CreateVectorDatabase.script

talkpipe-0.9.2/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_1_CreateVectorDatabase.sh → talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_1_CreateVectorDatabase.script

@@ -10,12 +10,8 @@
 # 3. The embeddings are stored in a vector index at the specified path.
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    INPUT FROM "../Tutorial_1-Document_Indexing/stories.json"
-    | readJsonl 
-    | progressTicks[tick_count=1, print_count=True] 
-    | llmEmbed[field="content", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | addVector[path="./vector_index", vector_field="vector", metadata_field_list="title,content", overwrite=True]
-'
-
-chatterlang_script --script CHATTERLANG_SCRIPT 
+INPUT FROM "../Tutorial_1-Document_Indexing/stories.json"
+| readJsonl
+| progressTicks[tick_count=1, print_count=True]
+| llmEmbed[field="content", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| addToLanceDB[path="./vector_index", table_name="stories", vector_field="vector", metadata_field_list="title,content", overwrite=True]

talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_1_CreateVectorDatabase.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_script --script Step_1_CreateVectorDatabase.script 

talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_2_SearchByExample.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script Step_2_SearchByExample.script

talkpipe-0.9.2/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_2_SearchByExample.sh → talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_2_SearchByExample.script

@@ -11,15 +11,11 @@
 #    working with the endpoint api because the input data will be sent to the
 #    browser.
 # 3. Use the `llmEmbed` segment to convert the example into a vector.
-# 4. Use the `searchVector` segment to search the vector index for similar documents.
+# 4. Use the `searchLanceDB` segment to search the vector index for similar documents.
 # 5. Format the results using `formatItem` and print them.
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='    
-    | copy
-    | llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | searchVector[vector_field="vector", path="./vector_index", top_k=10]
-    | formatItem[field_list="document.title:Title, document.content:Content, score:Score"]
-'
-
-chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script CHATTERLANG_SCRIPT
+| copy
+| llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| searchLanceDB[field="vector", path="./vector_index", table_name="stories", limit=10]
+| formatItem[field_list="document.title:Title, document.content:Content, score:Score"]

talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_2_SearchByExample.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script Step_2_SearchByExample.script

talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_3_SpecializedRag.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_serve --form-config story_by_example_ui.yml --load-module step_3_extras.py --display-property example --script Step_3_SpecializedRag.script