TalkPipe 0.9.2__tar.gz → 0.9.3__tar.gz

{talkpipe-0.9.2 → talkpipe-0.9.3}/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 0.9.3 
+### Improvements
+- Updated tutorials to use lancedb.  
+- Moved tutorial scripts out of shell scripts into their own files
+- Fixed issue with Docker container not including all optional installation components
+- Fixed string encoding issue with web page downloads
+- Updated documentation for plugins to fix error
+- Removed invalid script in pyproject.toml
+
 ## 0.9.2
 ### Improvements
 - Added ability for sources and segments to have multiple names in chatterlang.

{talkpipe-0.9.2 → talkpipe-0.9.3}/Dockerfile

@@ -34,8 +34,8 @@ COPY --chown=builder:builder src/ src/
 COPY --chown=builder:builder tests/ tests/
 
 # Install Python dependencies and build the package
-RUN python3 -m pip install --user --upgrade pip setuptools wheel build
-RUN python3 -m pip install --user numpy pandas matplotlib scikit-learn scipy
+RUN python3 -m pip install --upgrade pip setuptools wheel build
+RUN python3 -m pip install numpy pandas matplotlib scikit-learn scipy
 ENV SETUPTOOLS_SCM_PRETEND_VERSION_FOR_TALKPIPE=0.1.0
 RUN python3 -m pip install --user -e .[dev,all]
 RUN python3 -m pytest --log-cli-level=DEBUG
@@ -70,7 +70,8 @@ COPY --from=builder --chown=app:app /build/dist/*.whl /tmp/
 RUN python3 -m pip install --no-cache-dir --upgrade pip && \
     python3 -m pip install --no-cache-dir \
         numpy pandas matplotlib scikit-learn scipy && \
-    python3 -m pip install --no-cache-dir /tmp/*.whl && \
+    WHEEL_FILE=$(ls /tmp/*.whl) && \
+    python3 -m pip install --no-cache-dir "${WHEEL_FILE}[all]" && \
     rm -f /tmp/*.whl
 
 # Copy only necessary runtime files

{talkpipe-0.9.2 → talkpipe-0.9.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: talkpipe
-Version: 0.9.2
+Version: 0.9.3
 Summary: Python internal and external DSL for writing generative AI analytics
 Author-email: Travis Bauer <tlbauer@sandia.gov>
 Maintainer-email: Travis Bauer <tlbauer@sandia.gov>

{talkpipe-0.9.2 → talkpipe-0.9.3}/docs/api-reference/talkpipe-plugin-manager.md

@@ -4,7 +4,7 @@ Command-line tool for managing and inspecting TalkPipe plugins.
 
 ## Overview
 
-The `talkpipe_plugin_manager` command provides utilities to list, reload, and debug TalkPipe plugins that are installed via Python entry points.
+The `talkpipe_plugins` command provides utilities to list, reload, and debug TalkPipe plugins that are installed via Python entry points.
 
 ## Installation
 
@@ -21,7 +21,7 @@ pip install talkpipe
 Display all discovered plugins and their status:
 
 ```bash
-talkpipe_plugin_manager --list
+talkpipe_plugins --list
 ```
 
 **Example output:**
@@ -40,7 +40,7 @@ Failed plugins (1):
 Reload a specific plugin by name (useful during development):
 
 ```bash
-talkpipe_plugin_manager --reload my-data-plugin
+talkpipe_plugins --reload my-data-plugin
 ```
 
 ### Verbose Output
@@ -48,7 +48,7 @@ talkpipe_plugin_manager --reload my-data-plugin
 Show detailed information:
 
 ```bash
-talkpipe_plugin_manager --list --verbose
+talkpipe_plugins --list --verbose
 ```
 
 ## Plugin Discovery
@@ -65,7 +65,7 @@ setup(
     name="my-talkpipe-plugin",
     entry_points={
         "talkpipe.plugins": [
-            "my_plugin = my_plugin.plugin:NetworkPlugin",
+            "my_plugin = my_plugin.plugin",
         ]
     }
 )
@@ -74,21 +74,31 @@ setup(
 **pyproject.toml:**
 ```toml
 [project.entry-points."talkpipe.plugins"]
-my_plugin = "my_plugin.plugin:NetworkPlugin"
+my_plugin = "my_plugin.plugin"
 ```
 
+The entry point should reference the **module** that contains your plugin code. Component registration happens automatically through decorators when the module is imported.
+
 ## How Plugin Loading Works
 
 When TalkPipe discovers a plugin, it performs the following steps:
 
-1. **Import the module**: The plugin loader imports the module specified by the entry point (e.g., `my_plugin.plugin`)
+1. **Load the entry point**: The plugin loader loads whatever the entry point references (typically a module or function)
 2. **Automatic registration**: During import, any sources and segments defined or imported by the module are automatically registered with TalkPipe's registry system through decorators like `@registry.register_segment()`
-3. **Optional initialization**: If the loaded class/module has an `initialize_plugin` function or method, it will be called for additional setup
+3. **Optional initialization**: If the loaded object has an `initialize_plugin` function attribute, it will be called for additional setup
 
 The key insight is that **simply importing the plugin module triggers component registration**. This happens through TalkPipe's decorator-based registry system.
 
+**Important**: The entry point should reference either:
+- A **module** (e.g., `my_plugin.plugin`) that can have a module-level `initialize_plugin()` function
+- A **function** directly (e.g., `my_plugin.plugin:setup_plugin`)
+
+Entry points pointing to classes will have the `initialize_plugin` called on the **class itself** (not an instance), so it would need to be a static method or classmethod if using a class.
+
 ### Example Plugin Structure
 
+**Recommended: Module-level pattern**
+
 ```python
 # my_plugin/plugin.py
 from talkpipe.chatterlang import registry
@@ -104,19 +114,22 @@ class JsonParseSegment(AbstractSegment):
     """JSON parsing segment - registered automatically on import."""
     pass
 
-class NetworkPlugin:
-    """Main plugin class referenced by entry point."""
-    
-    def initialize_plugin(self):
-        """Optional: Called after module import for additional setup."""
-        print("Network plugin initialized!")
-        # Additional setup if needed
+def initialize_plugin():
+    """Optional: Called after module import for additional setup."""
+    print("Network plugin initialized!")
+    # Additional setup if needed
+```
+
+**Entry point configuration** (points to the module):
+```toml
+[project.entry-points."talkpipe.plugins"]
+network_plugin = "my_plugin.plugin"
 ```
 
-When this module is imported:
+When this module is loaded:
 1. The `@registry.register_source("httpGet")` decorator runs, registering the `HttpGetSource`
-2. The `@registry.register_segment("jsonParse")` decorator runs, registering the `JsonParseSegment`  
-3. The `NetworkPlugin.initialize_plugin()` method is called for any additional setup
+2. The `@registry.register_segment("jsonParse")` decorator runs, registering the `JsonParseSegment`
+3. The module-level `initialize_plugin()` function is called for any additional setup
 
 ### Alternative: Module-Level Registration
 
@@ -157,10 +170,10 @@ During plugin development, you can reload plugins without restarting Python:
 ```bash
 # Make changes to your plugin code
 # Then reload it
-talkpipe_plugin_manager --reload my-plugin
+talkpipe_plugins --reload my-plugin
 
 # Verify it loaded successfully
-talkpipe_plugin_manager --list
+talkpipe_plugins --list
 ```
 
 ## Integration with TalkPipe

{talkpipe-0.9.2 → talkpipe-0.9.3}/docs/tutorials/Tutorial_1-Document_Indexing/README.md

@@ -57,20 +57,22 @@ However, for testing and development purposes, we often need synthetic data that
 
 ### The Solution: AI-Generated Stories
 
-The first step uses TalkPipe's ChatterLang scripting language to generate 50 fictional stories about technology development. Here's what happens:
+The first step uses TalkPipe's ChatterLang scripting language to generate 50 fictional stories about technology development. The pipeline is defined in `Step_1_CreateSyntheticData.script`:
 
-```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    LOOP 50 TIMES {
-        INPUT FROM "Write a fictitious five sentence story about technology development in an imaginary country." 
-        | llmPrompt[source="ollama", model="llama3.2", multi_turn=False] 
-        | toDict[field_list="_:content"] 
-        | llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False] 
-        | dumpsJsonl | print;
-    }
-'
+```
+LOOP 50 TIMES {
+    INPUT FROM "Write a fictitious five sentence story about technology development in an imaginary country."
+    | llmPrompt[source="ollama", model="llama3.2", multi_turn=False]
+    | toDict[field_list="_:content"]
+    | llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False]
+    | dumpsJsonl | print;
+}
+```
+
+To run this script:
 
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT > stories.json
+```bash
+chatterlang_script --script Step_1_CreateSyntheticData.script > stories.json
 ```
 
 ### Breaking Down the Pipeline
@@ -166,17 +168,19 @@ Think of an index like the index at the back of a book, but much more sophistica
 
 ### The Solution: Whoosh Indexing
 
-Step 2 takes our generated stories and creates a searchable index using the Whoosh library:
+Step 2 takes our generated stories and creates a searchable index using the Whoosh library. The pipeline is defined in `Step_2_IndexStories.script`:
 
-```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    INPUT FROM "stories.json" 
-    | readJsonl 
-    | progressTicks[tick_count=1, print_count=True]
-    | indexWhoosh[index_path="./full_text_index", field_list="content,title", overwrite=True]
-'
+```
+INPUT FROM "stories.json"
+| readJsonl
+| progressTicks[tick_count=1, print_count=True]
+| indexWhoosh[index_path="./full_text_index", field_list="content,title", overwrite=True]
+```
+
+To run this script:
 
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT
+```bash
+chatterlang_script --script Step_2_IndexStories.script
 ```
 
 ### Understanding the Indexing Pipeline
@@ -283,15 +287,17 @@ Most search implementations require significant custom development, but TalkPipe
 
 ### The Solution: Dual Interface Search
 
-Step 3 creates both an API endpoint and a web interface using a single command:
+Step 3 creates both an API endpoint and a web interface using a single command. The pipeline is defined in `Step_3_SearchStories.script`:
 
-```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-  | searchWhoosh[index_path="full_text_index", field="query"] 
-  | formatItem[field_list="document.title:Title,document.content:Content,score:Score"]
-'
+```
+| searchWhoosh[index_path="full_text_index", field="query"]
+| formatItem[field_list="document.title:Title,document.content:Content,score:Score"]
+```
 
-python -m talkpipe.app.chatterlang_serve --form-config story_search_ui.yml --title "Story Search" --display-property query --script CHATTERLANG_SCRIPT
+To run this script:
+
+```bash
+chatterlang_serve --form-config story_search_ui.yml --title "Story Search" --display-property query --script Step_3_SearchStories.script
 ```
 
 ### Understanding the Search System

talkpipe-0.9.2/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.sh → talkpipe-0.9.3/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.script

@@ -17,14 +17,10 @@
 # 6. The output is redirected to a file named "stories.json".
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    LOOP 50 TIMES {
-        INPUT FROM "Write a fictitious five sentence story about technology development in an imaginary country." 
-        | llmPrompt[source="ollama", model="llama3.2", multi_turn=False] 
-        | toDict[field_list="_:content"] 
-        | llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False] 
-        | dumpsJsonl | print;
-    }
-'
-#chatterlang_script --script "
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT > stories.json
+LOOP 50 TIMES {
+    INPUT FROM "Write a fictitious five sentence story about technology development in an imaginary country."
+    | llmPrompt[source="ollama", model="llama3.2", multi_turn=False]
+    | toDict[field_list="_:content"]
+    | llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False]
+    | dumpsJsonl | print;
+}

talkpipe-0.9.3/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_script --script Step_1_CreateSyntheticData.script > stories.json

talkpipe-0.9.2/docs/tutorials/Tutorial_1-Document_Indexing/Step_2_IndexStories.sh → talkpipe-0.9.3/docs/tutorials/Tutorial_1-Document_Indexing/Step_2_IndexStories.script

@@ -6,8 +6,8 @@
 # The indexed data can then be used for full-text search.
 #
 # As a side note, the first half of this script issues a single piece of data, the
-# filename.  The next segment, `readJsonl`, reads the JSONL file line by line and 
-# issues one decoded JSON object at a time.  This is a good example of how the 
+# filename.  The next segment, `readJsonl`, reads the JSONL file line by line and
+# issues one decoded JSON object at a time.  This is a good example of how the
 # constitution of the data being processed can change as it flows through the pipeline.
 #
 # The pipeline is:
@@ -16,12 +16,7 @@
 # 3. The index is stored in the specified path "./full_text_index".
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    INPUT FROM "stories.json" 
-    | readJsonl 
-    | progressTicks[tick_count=1, print_count=True]
-    | indexWhoosh[index_path="./full_text_index", field_list="content,title", overwrite=True]
-'
-
-#chatterlang_script --script "
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT
+INPUT FROM "stories.json"
+| readJsonl
+| progressTicks[tick_count=1, print_count=True]
+| indexWhoosh[index_path="./full_text_index", field_list="content,title", overwrite=True]

talkpipe-0.9.3/docs/tutorials/Tutorial_1-Document_Indexing/Step_2_IndexStories.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_script --script Step_2_IndexStories.script

talkpipe-0.9.2/docs/tutorials/Tutorial_1-Document_Indexing/Step_3_SearchStories.sh → talkpipe-0.9.3/docs/tutorials/Tutorial_1-Document_Indexing/Step_3_SearchStories.script

@@ -12,8 +12,5 @@
 # needing to write any code.
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='
-  | searchWhoosh[index_path="full_text_index", field="query"] 
-  | formatItem[field_list="document.title:Title,document.content:Content,score:Score"]
-'
-chatterlang_serve --form-config story_search_ui.yml --title \"Story\ Search\" --display-property query --script CHATTERLANG_SCRIPT
+| searchWhoosh[index_path="full_text_index", field="query"]
+| formatItem[field_list="document.title:Title,document.content:Content,score:Score"]

talkpipe-0.9.3/docs/tutorials/Tutorial_1-Document_Indexing/Step_3_SearchStories.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_serve --form-config story_search_ui.yml --title "Story Search" --display-property query --script Step_3_SearchStories.script

{talkpipe-0.9.2 → talkpipe-0.9.3}/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/README.md

@@ -34,16 +34,20 @@ Vector embeddings solve this by converting text into high-dimensional mathematic
 
 ### The Implementation
 
+The pipeline is defined in `Step_1_CreateVectorDatabase.script`:
+
+```
+INPUT FROM "../Tutorial_1-Document_Indexing/stories.json"
+| readJsonl
+| progressTicks[tick_count=1, print_count=True]
+| llmEmbed[field="content", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| addToLanceDB[path="./vector_index", table_name="stories", vector_field="vector", metadata_field_list="title,content", overwrite=True]
+```
+
+To run this script:
+
 ```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    INPUT FROM "../Tutorial_1-Document_Indexing/stories.json"
-    | readJsonl 
-    | progressTicks[tick_count=1, print_count=True] 
-    | llmEmbed[field="content", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | addVector[path="./vector_index", vector_field="vector", metadata_field_list="title,content", overwrite=True]
-'
-
-python -m talkpipe.app.chatterlang_script --script CHATTERLANG_SCRIPT
+chatterlang_script --script Step_1_CreateVectorDatabase.script
 ```
 
 ### Breaking Down the Pipeline
@@ -67,12 +71,12 @@ The `mxbai-embed-large` model is specifically designed for semantic search - it'
 
 **3. Building the Index**
 ```
-| addVector[path="./vector_index", vector_field="vector", metadata_field_list="title,content", overwrite=True]
+| addToLanceDB[path="./vector_index", table_name="stories", vector_field="vector", metadata_field_list="title,content", overwrite=True]
 ```
 This creates a specialized index that:
-- Stores vectors for similarity search
+- Stores vectors for similarity search in a LanceDB table named "stories"
 - Preserves original metadata (title and content) for retrieval
-- Enables fast nearest-neighbor queries
+- Enables fast nearest-neighbor queries using LanceDB's efficient vector search capabilities
 
 ### Real-World Applications
 
@@ -96,15 +100,19 @@ Your users don't always know the right keywords. Sometimes they have an example
 
 ### The Solution: Semantic Search Interface
 
+The pipeline is defined in `Step_2_SearchByExample.script`:
+
+```
+| copy
+| llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| searchLanceDB[field="vector", path="./vector_index", table_name="stories", limit=10]
+| formatItem[field_list="document.title:Title, document.content:Content, score:Score"]
+```
+
+To run this script:
+
 ```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='    
-    | copy
-    | llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | searchVector[vector_field="vector", path="./vector_index"]
-    | formatItem[field_list="document.title:Title, document.content:Content, score:Score"]
-'
-
-python -m talkpipe.app.chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script CHATTERLANG_SCRIPT
+chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script Step_2_SearchByExample.script
 ```
 
 ### Understanding the Search Pipeline
@@ -123,9 +131,9 @@ The user's example text is converted to a vector using the same model that index
 
 **3. Vector Search**
 ```
-| searchVector[vector_field="vector", path="./vector_index"]
+| searchLanceDB[field="vector", path="./vector_index", table_name="stories", limit=10]
 ```
-This finds the documents whose vectors are closest to the query vector - literally the nearest neighbors in high-dimensional space.
+This finds the documents whose vectors are closest to the query vector - literally the nearest neighbors in high-dimensional space. LanceDB provides efficient approximate nearest neighbor search for fast retrieval.
 
 **4. Result Formatting**
 ```
@@ -167,23 +175,27 @@ Finding relevant documents is helpful, but what users often really want is an an
 
 ### The RAG Implementation
 
+The pipeline is defined in `Step_3_SpecializedRag.script`:
+
+```
+| copy
+| llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| searchLanceDB[field="vector", path="./vector_index", table_name="stories", all_results_at_once=True, set_as="results"]
+| ragPrompt
+| llmPrompt[source="ollama", model="llama3.2"]
+```
+
+To run this script:
+
 ```bash
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    | copy
-    | llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | searchVector[vector_field="vector", path="./vector_index", all_results_at_once=True, set_as="results"]
-    | ragPrompt
-    | llmPrompt[source="ollama", model="llama3.2"]
-'
-
-python -m talkpipe.app.chatterlang_serve --form-config story_by_example_ui.yml --load-module step_3_extras.py --display-property example --script CHATTERLANG_SCRIPT
+chatterlang_serve --form-config story_by_example_ui.yml --load-module step_3_extras.py --display-property example --script Step_3_SpecializedRag.script
 ```
 
 ### What's Different in the RAG Pipeline
 
 **1. Batch Results Collection**
 ```
-| searchVector[..., all_results_at_once=True, set_as="results"]
+| searchLanceDB[..., all_results_at_once=True, set_as="results"]
 ```
 Instead of processing results one by one, we collect all search results together. This allows the next step to see the full context.
 

talkpipe-0.9.2/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_1_CreateVectorDatabase.sh → talkpipe-0.9.3/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_1_CreateVectorDatabase.script

@@ -10,12 +10,8 @@
 # 3. The embeddings are stored in a vector index at the specified path.
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    INPUT FROM "../Tutorial_1-Document_Indexing/stories.json"
-    | readJsonl 
-    | progressTicks[tick_count=1, print_count=True] 
-    | llmEmbed[field="content", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | addVector[path="./vector_index", vector_field="vector", metadata_field_list="title,content", overwrite=True]
-'
-
-chatterlang_script --script CHATTERLANG_SCRIPT 
+INPUT FROM "../Tutorial_1-Document_Indexing/stories.json"
+| readJsonl
+| progressTicks[tick_count=1, print_count=True]
+| llmEmbed[field="content", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| addToLanceDB[path="./vector_index", table_name="stories", vector_field="vector", metadata_field_list="title,content", overwrite=True]

talkpipe-0.9.3/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_1_CreateVectorDatabase.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_script --script Step_1_CreateVectorDatabase.script 

talkpipe-0.9.2/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_2_SearchByExample.sh → talkpipe-0.9.3/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_2_SearchByExample.script

@@ -11,15 +11,11 @@
 #    working with the endpoint api because the input data will be sent to the
 #    browser.
 # 3. Use the `llmEmbed` segment to convert the example into a vector.
-# 4. Use the `searchVector` segment to search the vector index for similar documents.
+# 4. Use the `searchLanceDB` segment to search the vector index for similar documents.
 # 5. Format the results using `formatItem` and print them.
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='    
-    | copy
-    | llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | searchVector[vector_field="vector", path="./vector_index", top_k=10]
-    | formatItem[field_list="document.title:Title, document.content:Content, score:Score"]
-'
-
-chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script CHATTERLANG_SCRIPT
+| copy
+| llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| searchLanceDB[field="vector", path="./vector_index", table_name="stories", limit=10]
+| formatItem[field_list="document.title:Title, document.content:Content, score:Score"]

talkpipe-0.9.3/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_2_SearchByExample.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script Step_2_SearchByExample.script

talkpipe-0.9.2/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_3_SpecializedRag.sh → talkpipe-0.9.3/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_3_SpecializedRag.script

@@ -3,22 +3,18 @@
 # It allows users to input an example query, which is then converted into a vector
 # using an LLM embedding model. The vector is used to search a vector index for similar
 # documents, and the results are formatted and returned.
-# 
+#
 # The pipeline is:
 # 1. Read the example query from the user.
 # 2. Make a defensive copy of the input data for browser compatibility.
 # 3. Use the `llmEmbed` segment to convert the example into a vector.
-# 4. Use the `searchVector` segment to search the vector index for similar documents
+# 4. Use the `searchLanceDB` segment to search the vector index for similar documents
 # 5. use ragPrompt to create a prompt from the query and the retrieve documents.
 # 6. Use the `llmPrompt` segment to generate a final response using the LLM.
 ###################################################################################
 
-export TALKPIPE_CHATTERLANG_SCRIPT='
-    | copy
-    | llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
-    | searchVector[vector_field="vector", path="./vector_index", all_results_at_once=True, set_as="results"]
-    | ragPrompt
-    | llmPrompt[source="ollama", model="llama3.2"]
-'
-
-chatterlang_serve --form-config story_by_example_ui.yml --load-module step_3_extras.py --display-property example --script CHATTERLANG_SCRIPT
+| copy
+| llmEmbed[field="example", source="ollama", model="mxbai-embed-large", set_as="vector"]
+| searchLanceDB[field="vector", path="./vector_index", table_name="stories", all_results_at_once=True, set_as="results"]
+| ragPrompt
+| llmPrompt[source="ollama", model="llama3.2"]

talkpipe-0.9.3/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_3_SpecializedRag.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_serve --form-config story_by_example_ui.yml --load-module step_3_extras.py --display-property example --script Step_3_SpecializedRag.script