TalkPipe 0.9.3__tar.gz → 0.9.4__tar.gz

{talkpipe-0.9.3 → talkpipe-0.9.4}/.github/workflows/ci-cd.yml

@@ -148,7 +148,7 @@ jobs:
         output: 'trivy-results.sarif'
     
     - name: Upload Trivy scan results to GitHub Security tab
-      uses: github/codeql-action/upload-sarif@v3
+      uses: github/codeql-action/upload-sarif@v4
       if: always()
       with:
         sarif_file: 'trivy-results.sarif'
@@ -171,15 +171,15 @@ jobs:
       uses: actions/checkout@v5
     
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v3
+      uses: github/codeql-action/init@v4
       with:
         languages: ${{ matrix.language }}
     
     - name: Autobuild
-      uses: github/codeql-action/autobuild@v3
+      uses: github/codeql-action/autobuild@v4
     
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v3
+      uses: github/codeql-action/analyze@v4
 
   publish-package:
     runs-on: ubuntu-latest

{talkpipe-0.9.3 → talkpipe-0.9.4}/CHANGELOG.md

@@ -1,5 +1,43 @@
 # Changelog
 
+## 0.9.4
+### Improvements
+- Changes several logging statements to make the consistent with the rest of the code
+- Refactored **lambda** to use AbstractFieldSegment for consistency
+- Removed fail_on_error parameter in **lambda** and **lambdaFilter**.  This is an API breaking change.
+  The segments will now always fail on an error and there is no option to silently fail.
+- Changed chatterlang_serve so that it raises and exception an exits if given a script that can't be
+  compiled.  The previous behavior was that it would issue a log message and fall back to the default
+  script.
+- Added unit test for parse_unknown_args and expanded it to support boolean flag parameters
+- Marked UMAP as deprecated.  It will be removed in 1.0.  It requires additional dependencies that no
+  other core modules need.
+- Updated the ollama embedding and chat connector so that it uses the OLLAMA_SERVER_URL configuration
+  variable as the host where ollama is installed.  So OLLAMA_SERVER_URL can be set in the TOML configuration
+  file or TALKPIPE_OLLAMA_SERVER_URL can be set as an environment variable.
+- Implemented better compile errors for when sources or segments are not found.  It had been a key error.
+  It will now be a compile error.
+- Added batch files for the tutorials so it is easier to run them on Windows
+- Enhanced **prompt** source to catch downstream pipeline errors and continue prompting instead of
+  crashing. When a user enters invalid input that causes an exception in downstream segments, the error
+  is displayed with a full traceback, and the prompt continues accepting input. This makes interactive
+  pipelines more robust and user-friendly. The error-resilient behavior is enabled by default but can
+  be disabled by setting `error_resilient=False`.
+- Updated **chatterlang_serve** to support API key configuration via the configuration system. When
+  `--api-key` is not specified on the command line, it will check for `API_KEY` in the configuration,
+  which automatically checks the `TALKPIPE_API_KEY` environment variable. This makes it easier to
+  deploy chatterlang_serve in Docker and CI/CD environments without exposing API keys in command history.
+- Added support for single-quoted strings in ChatterLang scripts. Single quotes work identically to
+  double quotes, including support for escaping quotes by doubling them (`''` → `'`). This provides
+  more flexibility when writing scripts, especially when the string content contains double quotes.
+- Enhanced **echo** source with an `n` parameter to repeat the output data multiple times. For example,
+  `echo[data='a,b', n=3]` will emit: a, b, a, b, a, b. This is useful for testing pipelines and
+  generating repeated test data.
+- Improved **chatterlang_serve** startup output to clearly distinguish between the user interface URL
+  and the API endpoint URL. The server now displays a formatted message showing both URLs with clear
+  labels when it starts.
+- Updated tutorial 1, step 1 to show more feedback to the user so it doesn't just seem to be hanging
+
 ## 0.9.3 
 ### Improvements
 - Updated tutorials to use lancedb.  

{talkpipe-0.9.3 → talkpipe-0.9.4}/CLAUDE.md

@@ -16,6 +16,8 @@ ChatterLang is an external DSL and is defined under talkpipe/chatterlang
 Other packages in the project other than util include implementations of pipe sources and segments
 Util contains various generally usable utility functions, expecially for basic data manipulation and configuration.
 talkpipe/README.md contains a longer overview
+After making a change, update CHANGELOG.md
+When making a change, write a unit test that fails without the change, verify that the unit test fails, make the change and then verity that the unit test now passes.  Notify the user if it is not reasonable to write the unit test.
 
 ## Important Commands
 python -m build for creating a new release

{talkpipe-0.9.3 → talkpipe-0.9.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: talkpipe
-Version: 0.9.3
+Version: 0.9.4
 Summary: Python internal and external DSL for writing generative AI analytics
 Author-email: Travis Bauer <tlbauer@sandia.gov>
 Maintainer-email: Travis Bauer <tlbauer@sandia.gov>

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_script --script Step_1_CreateSyntheticData.script

{talkpipe-0.9.3 → talkpipe-0.9.4}/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.script

@@ -17,10 +17,13 @@
 # 6. The output is redirected to a file named "stories.json".
 ###################################################################################
 
-LOOP 50 TIMES {
-    INPUT FROM "Write a fictitious five sentence story about technology development in an imaginary country."
-    | llmPrompt[source="ollama", model="llama3.2", multi_turn=False]
-    | toDict[field_list="_:content"]
-    | llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False]
-    | dumpsJsonl | print;
-}
+
+INPUT FROM "Generating 50 synthetic stories into stories.json" | print;
+INPUT FROM echo[data="Write a fictitious five sentence story about technology development in an imaginary country.", n=50]
+| llmPrompt[source="ollama", model="llama3.2", multi_turn=False]
+| toDict[field_list="_:content"]
+| llmPrompt[source="ollama", model="llama3.2", system_prompt="Write exactly one title for this story in plain text with no markdown", field="content", set_as="title", multi_turn=False]
+| progressTicks[tick_count=1, eol_count=10, print_count=true]
+| dumpsJsonl 
+| writeString[fname="stories.json"];
+

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_1_CreateSyntheticData.sh

@@ -0,0 +1,2 @@
+#!/bin/bash
+chatterlang_script --script Step_1_CreateSyntheticData.script 

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_2_IndexStories.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_script --script Step_2_IndexStories.script

talkpipe-0.9.4/docs/tutorials/Tutorial_1-Document_Indexing/Step_3_SearchStories.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_serve --form-config story_search_ui.yml --title "Story Search" --display-property query --script Step_3_SearchStories.script

talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_1_CreateVectorDatabase.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_script --script Step_1_CreateVectorDatabase.script

talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_2_SearchByExample.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_serve --form-config story_by_example_ui.yml --display-property example --script Step_2_SearchByExample.script

talkpipe-0.9.4/docs/tutorials/Tutorial_2-Search_by_Example_and_RAG/Step_3_SpecializedRag.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_serve --form-config story_by_example_ui.yml --load-module step_3_extras.py --display-property example --script Step_3_SpecializedRag.script

talkpipe-0.9.4/docs/tutorials/Tutorial_3_Report_Writing/Step_1_ExecutiveSummaryGeneration.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_serve --form-config report_topic_ui.yml --load-module step_1_extras.py --display-property topic --script Step_1_ExecutiveSummaryGeneration.script

talkpipe-0.9.4/docs/tutorials/Tutorial_3_Report_Writing/Step_2_DetailedAnalysisReportGeneration.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_serve --form-config report_topic_ui.yml --load-module step_2_extras.py --display-property topic --script Step_2_DetailedAnalysisReportGeneration.script

talkpipe-0.9.4/docs/tutorials/Tutorial_3_Report_Writing/Step_3_MultiFormatReportGeneration.bat

@@ -0,0 +1,2 @@
+@echo off
+chatterlang_serve --form-config multi_format_ui.yml --load-module step3_extras.py --display-property topic --script Step_3_MultiFormatReportGeneration.script

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/app/chatterlang_serve.py

@@ -51,16 +51,10 @@ class UserSession:
     
     def compile_script(self, script_content: str):
         """Compile a Chatterlang script for this session"""
-        try:
-            from talkpipe.chatterlang import compile as chatterlang_compile
-            
-            # Compile script - configuration values are accessible via $key syntax
-            self.compiled_script = chatterlang_compile(script_content)
-            self.compiled_script = self.compiled_script.as_function(single_in=True, single_out=False)
-            logger.info(f"Session {self.session_id}: Script compiled successfully")
-        except Exception as e:
-            logger.error(f"Session {self.session_id}: Error compiling script: {e}")
-            self.compiled_script = None
+        # Compile script - configuration values are accessible via $key syntax
+        self.compiled_script = compile(script_content)
+        self.compiled_script = self.compiled_script.as_function(single_in=True, single_out=False)
+        logger.info(f"Session {self.session_id}: Script compiled successfully")
     
     def add_to_history(self, entry: dict):
         """Add entry to session history"""
@@ -1652,16 +1646,21 @@ class ChatterlangServer:
     
     def start(self, background: bool = False):
         """Start the FastAPI server"""
-        logger.info(f"Starting JSON Data Receiver on http://{self.host}:{self.port}")
-        logger.info(f"Main interface: http://{self.host}:{self.port}/")
-        logger.info(f"Streaming interface: http://{self.host}:{self.port}/stream")
-        logger.info(f"API documentation: http://{self.host}:{self.port}/docs")
+        print(f"\n{'='*60}")
+        print(f"ChatterLang Server Started")
+        print(f"{'='*60}")
+        print(f"User Interface:     http://{self.host}:{self.port}/stream")
+        print(f"API Endpoint:       http://{self.host}:{self.port}/process")
+        print(f"API Documentation:  http://{self.host}:{self.port}/docs")
         if self.require_auth:
-            logger.info(f"Port {self.port}: API key authentication: ENABLED")
-        
-        processor_name = self.processor_function.__name__ 
+            print(f"Authentication:     ENABLED (API key required)")
+        print(f"{'='*60}\n")
+
+        # Also log for debugging purposes
+        logger.info(f"Starting JSON Data Receiver on http://{self.host}:{self.port}")
+        processor_name = self.processor_function.__name__
         logger.info(f"Port {self.port}: Using processor function: {processor_name}")
-        
+
         if background:
             self.server_thread = threading.Thread(
                 target=lambda: uvicorn.run(self.app, host=self.host, port=self.port),
@@ -1794,19 +1793,24 @@ def go():
                         help='Property of the input json to display in the stream interface as user input.')
 
     args, unknown_args = parser.parse_known_args()
-    
+
     # Parse unknown arguments and add to configuration so they're accessible via $key syntax
     constants = parse_unknown_args(unknown_args)
-    
+
     # Add command-line constants to the configuration
     if constants:
         add_config_values(constants, override=True)
         print(f"Added command-line values to configuration: {list(constants.keys())}")
-    
+
     if args.load_module:
         for module_file in args.load_module:
             load_module_file(fname=module_file, fail_on_missing=False)
 
+    # Get API key from command line, or fall back to configuration (which checks environment variable)
+    api_key = args.api_key
+    if api_key is None:
+        api_key = get_config().get('API_KEY')
+
     script_content = None
     if args.script:
         script_content = load_script(args.script)
@@ -1829,7 +1833,7 @@ def go():
     receiver = ChatterlangServer(
         host=args.host,
         port=args.port,
-        api_key=args.api_key,
+        api_key=api_key,
         require_auth=args.require_auth,
         title=args.title,
         form_config=form_config,

talkpipe-0.9.4/src/talkpipe/app/static/unit-docs.html

@@ -0,0 +1 @@
+Documentation generation failed: Command '['/tmp/build-env-6ktuidus/bin/python', 'src/talkpipe/app/chatterlang_reference_generator.py', 'src', 'src/talkpipe/app/static/unit-docs.html', 'src/talkpipe/app/static/unit-docs.txt']' returned non-zero exit status 1.

talkpipe-0.9.4/src/talkpipe/app/static/unit-docs.txt

@@ -0,0 +1 @@
+Documentation generation failed: Command '['/tmp/build-env-6ktuidus/bin/python', 'src/talkpipe/app/chatterlang_reference_generator.py', 'src', 'src/talkpipe/app/static/unit-docs.html', 'src/talkpipe/app/static/unit-docs.txt']' returned non-zero exit status 1.

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/chatterlang/compiler.py

@@ -16,6 +16,10 @@ from talkpipe.pipe import io
 
 logger = logging.getLogger(__name__)
 
+class CompileError(Exception):
+    """ Exception raised when a compilation error occurs in chatterlang """
+    pass
+
 @singledispatch
 def compile(script: ParsedScript, runtime: RuntimeComponent = None) -> Callable:
     """ Compile a parsed script into a callable function 
@@ -58,7 +62,10 @@ def _(pipeline: ParsedPipeline, runtime: RuntimeComponent) -> Pipeline:
             ans = VariableSource(pipeline.input_node.source.name)
             logger.debug(f"Created variable source with name {pipeline.input_node.source.name}")
         else:
-            ans = registry.input_registry.get(pipeline.input_node.source.name)(**_resolve_params(pipeline.input_node.params, runtime=runtime))
+            try:
+                ans = registry.input_registry.get(pipeline.input_node.source.name)(**_resolve_params(pipeline.input_node.params, runtime=runtime))
+            except KeyError:
+                raise CompileError(f"Source '{pipeline.input_node.source.name}' not found")
             logger.debug(f"Created registered input {pipeline.input_node.source.name}")
         ans.runtime = runtime
 
@@ -68,7 +75,10 @@ def _(pipeline: ParsedPipeline, runtime: RuntimeComponent) -> Pipeline:
             next_transform = VariableSetSegment(transform.name)
             logger.debug(f"Created variable set segment for {transform.name}")
         elif isinstance(transform, SegmentNode):
-            next_transform = registry.segment_registry.get(transform.operation.name)(**_resolve_params(transform.params, runtime=runtime))
+            try:
+                next_transform = registry.segment_registry.get(transform.operation.name)(**_resolve_params(transform.params, runtime=runtime))
+            except KeyError:
+                raise CompileError(f"Segment '{transform.operation.name}' not found")
             logger.debug(f"Created segment {transform.operation.name}")
         elif isinstance(transform, ForkNode):
             next_transform = compile(transform, runtime)

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/chatterlang/parsers.py

@@ -133,11 +133,12 @@ identifier = regex(r'[a-zA-Z][a-zA-Z0-9_]*').map(lambda s: Identifier(name=s))
 variable = (string('@') >> identifier).map(lambda x: x).map(lambda x: VariableName(name=x.name))
 """A parser for variable names.  Variables are used to store and retrieve data in the pipeline."""
 environmentVariable = (string('$') >> identifier).map(lambda x: get_config()[x.name])
-# quoted_string = regex(r'"[^"]*"').map(lambda s: s[1:-1])
-quoted_string = regex(r'"(?:[^"]|"")*"').map(
-    lambda s: s[1:-1].replace('""', '"')
+# quoted_string supports both double and single quotes with escaping
+quoted_string = (
+    regex(r'"(?:[^"]|"")*"').map(lambda s: s[1:-1].replace('""', '"'))
+    | regex(r"'(?:[^']|'')*'").map(lambda s: s[1:-1].replace("''", "'"))
 )
-"""A parser for quoted strings."""
+"""A parser for quoted strings. Supports both double quotes ("...") and single quotes ('...')."""
 number = regex(r'-?\d+(\.\d+)?').map(lambda s: int(s) if '.' not in s else float(s))
 """A parser for numbers.  Returns both ints and floats."""
 

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/data/extraction.py

@@ -10,6 +10,8 @@ from pathlib import Path
 import glob
 import os
 
+logger = logging.getLogger(__name__)
+
 @register_segment("readtxt")
 @field_segment()
 def readtxt(file_path: Annotated[str, "Path to the text file to read"]):
@@ -30,15 +32,15 @@ def readtxt(file_path: Annotated[str, "Path to the text file to read"]):
     p = Path(file_path)
 
     if not p.exists():
-        logging.error(f"Path does not exist: {file_path}")
+        logger.error(f"Path does not exist: {file_path}")
         raise FileNotFoundError(f"Path does not exist: {file_path}")
 
     if p.is_file():
-        logging.info(f"Reading text file: {p}")
+        logger.debug(f"Reading text file: {p}")
         with p.open("r") as file:
             return file.read()
     else:
-        logging.error(f"Unsupported path type: {file_path}")
+        logger.error(f"Unsupported path type: {file_path}")
         raise FileNotFoundError(f"Unsupported path type: {file_path}")
         
 @register_segment("readdocx")
@@ -60,18 +62,18 @@ def readdocx(file_path: Annotated[str, "Path to the .docx file to read"]):
     p = Path(file_path)
 
     if not p.exists():
-        logging.error(f"Path does not exist: {file_path}")
+        logger.error(f"Path does not exist: {file_path}")
         raise FileNotFoundError(f"Path does not exist: {file_path}")
 
     if p.is_file():
-        logging.info(f"Reading docx file: {p}")
+        logger.info(f"Reading docx file: {p}")
         doc = Document(p)
         full_text = []
         for para in doc.paragraphs:
             full_text.append(para.text)
         return " ".join(full_text)
     else:
-        logging.error(f"Unsupported path type: {file_path}")
+        logger.error(f"Unsupported path type: {file_path}")
         raise FileNotFoundError(f"Unsupported path type: {file_path}")
 
 @register_segment("listFiles")
@@ -93,8 +95,8 @@ def listFiles(patterns: Annotated[Iterable[str], "Iterable of file patterns or p
         path = Path(expanded_pattern)
         if path.is_dir() and not any(char in expanded_pattern for char in ['*', '?', '[']):
             expanded_pattern = os.path.join(expanded_pattern, '*')
-        
-        logging.info(f"Searching for files matching pattern: {expanded_pattern}")
+
+        logger.info(f"Searching for files matching pattern: {expanded_pattern}")
         matches = glob.glob(expanded_pattern, recursive=True)
         
         for match in sorted(matches):
@@ -105,7 +107,7 @@ def listFiles(patterns: Annotated[Iterable[str], "Iterable of file patterns or p
                 else:
                     yield path.name
             else:
-                logging.debug(f"Skipping non-file: {match}")
+                logger.debug(f"Skipping non-file: {match}")
 
 @register_segment("readFile")
 class ReadFile(AbstractFieldSegment):
@@ -128,21 +130,21 @@ class ReadFile(AbstractFieldSegment):
 
     def __init__(self, field: str = None, set_as: str = None):
         super().__init__(field=field, set_as=set_as)
-        logging.debug("Initializing FileExtractor")
+        logger.debug("Initializing FileExtractor")
         self._extractors = {}
         self.register_extractor("txt", readtxt())
         self.register_extractor("md", readtxt())
         self.register_extractor("docx", readdocx())
 
     def register_extractor(self, file_extension:str, extractor):
-        logging.debug(f"Registering extractor for extension: {file_extension}")
+        logger.debug(f"Registering extractor for extension: {file_extension}")
         self._extractors[file_extension] = extractor
 
     def process_value(self, file_path:Union[str, PosixPath]):
         file_extension = file_path.split(".")[-1] if isinstance(file_path, str) else file_path.suffix[1:]
         if file_extension not in self._extractors:
-            logging.error(f"Unsupported file extension: {file_extension}")
+            logger.error(f"Unsupported file extension: {file_extension}")
             raise Exception(f"File extension {file_extension} not supported")
-        logging.debug(f"Extracting content from file: {file_path}")
+        logger.debug(f"Extracting content from file: {file_path}")
         return next(self._extractors[file_extension]([file_path]))
 

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/llm/chat.py

@@ -9,7 +9,7 @@ from abc import ABC, abstractmethod
 import logging
 from pydantic import BaseModel
 
-from talkpipe.llm.config import TALKPIPE_MODEL_NAME, TALKPIPE_SOURCE
+from talkpipe.util.constants import TALKPIPE_MODEL_NAME, TALKPIPE_SOURCE
 from talkpipe.util.data_manipulation import extract_property
 
 
@@ -31,6 +31,11 @@ class LLMPrompt(AbstractSegment):
     and TALKPIPE_default_source).  If those are not set, the values will be loaded
     from the configuration file (~/.talkpipe.toml).  If none of those are set, an 
     error will be raised.
+
+    Currently supported sources are "ollama," "openai," and "anthropic."  If 
+    you specify "ollama," you can optionally set the OLLAMA_SERVER_URL environment
+    variable or configuration value to point to a different server.  By default,
+    ollama assumes localhost.
     """
 
     def __init__(

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/llm/config.py

@@ -34,5 +34,3 @@ def getEmbeddingSources()->List[str]:
     return list(_embeddingAdapter.keys())
 
 
-TALKPIPE_MODEL_NAME = "default_model_name"
-TALKPIPE_SOURCE = "default_model_source"

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/llm/embedding_adapters.py

@@ -1,6 +1,7 @@
 from typing import List
-
 import numpy as np
+from talkpipe.util.config import get_config
+from talkpipe.util.constants import OLLAMA_SERVER_URL
 
 class AbstractEmbeddingAdapter:
     """Abstract class for embedding text.
@@ -44,8 +45,9 @@ class AbstractEmbeddingAdapter:
 class OllamaEmbedderAdapter(AbstractEmbeddingAdapter):
     """Embedding adapter for Ollama"""
 
-    def __init__(self, model: str):
+    def __init__(self, model: str, server_url: str = None):
         super().__init__(model, "ollama")
+        self._server_url = server_url
 
     def execute(self, text: str) -> List[float]:
         try:
@@ -55,7 +57,11 @@ class OllamaEmbedderAdapter(AbstractEmbeddingAdapter):
                 "Ollama is not installed. Please install it with: pip install talkpipe[ollama]"
             )
 
-        response = ollama.embed(
+        server_url = self._server_url
+        if not server_url:
+            server_url = get_config().get(OLLAMA_SERVER_URL, None)
+        client = ollama.Client(server_url) if server_url else ollama
+        response = client.embed(
             model=self.model_name,
             input=text
         )

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/llm/prompt_adapters.py

@@ -1,8 +1,9 @@
+from abc import ABC, abstractmethod
+import logging
 import json
 from pydantic import BaseModel
-import logging
-
-from abc import ABC, abstractmethod
+from talkpipe.util.config import get_config
+from talkpipe.util.constants import OLLAMA_SERVER_URL
 
 logger = logging.getLogger(__name__)
 
@@ -73,13 +74,25 @@ class AbstractLLMPromptAdapter(ABC):
 class OllamaPromptAdapter(AbstractLLMPromptAdapter):
     """Prompt adapter for Ollama
 
+    Note: By default, ollama assumes localhost for the ollama server.  
+    If your server is running elsewhere, you can set the OLLAMA_SERVER_URL 
+    environment variable or in the configuration, or pass the server_url 
+    parameter.
+
     """
 
-    def __init__(self, model: str, system_prompt: str = "You are a helpful assistant.", multi_turn: bool = True, temperature: float = None, output_format: BaseModel = None):
+    def __init__(self, 
+                 model: str, 
+                 system_prompt: str = "You are a helpful assistant.", 
+                 multi_turn: bool = True, 
+                 temperature: float = None, 
+                 output_format: BaseModel = None, 
+                 server_url: str = None):
         super().__init__(model, "ollama", system_prompt, multi_turn, temperature, output_format)
         # Ollama uses 0.5 as default when temperature is not specified
         if self._temperature is None:
             self._temperature = 0.5
+        self._server_url = server_url
 
     def execute(self, prompt: str) -> str:
         """Execute the chat model.
@@ -97,7 +110,11 @@ class OllamaPromptAdapter(AbstractLLMPromptAdapter):
         self._messages.append({"role": "user", "content": prompt})
 
         logger.debug(f"Sending chat request to Ollama model {self._model_name}")
-        response = ollama.chat(
+        server_url = self._server_url
+        if not server_url:
+            server_url = get_config().get(OLLAMA_SERVER_URL, None)
+        client = ollama.Client(server_url) if server_url else ollama
+        response = client.chat(
             self._model_name,
             messages=[self._system_message] + self._messages,
             format=self._output_format.model_json_schema() if self._output_format else None,

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/operations/matrices.py

@@ -1,3 +1,4 @@
+from deprecated import deprecated
 from typing import Optional, Annotated
 import numpy as np
 import umap
@@ -5,6 +6,7 @@ from sklearn.manifold import TSNE
 from talkpipe import register_segment, AbstractSegment
 
 @register_segment("reduceUMAP")
+@deprecated(reason="This segment is deprecated and will be removed in version 1.0.")
 class ReduceUMAP(AbstractSegment):
     """Use UMAP to reduce dimensionality of provided matrix.
     

{talkpipe-0.9.3 → talkpipe-0.9.4}/src/talkpipe/pipe/basic.py

@@ -13,7 +13,7 @@ from talkpipe.util.data_manipulation import extract_property, extract_template_f
 from talkpipe.util.data_manipulation import compileLambda
 from talkpipe.util.os import run_command
 from talkpipe.util.data_manipulation import get_type_safely
-from talkpipe.pipe.core import AbstractSegment, source, segment, field_segment
+from talkpipe.pipe.core import AbstractSegment, AbstractFieldSegment, source, segment, field_segment
 import talkpipe.chatterlang.registry as registry
 from talkpipe.util.data_manipulation import fill_template, dict_to_text, toDict
 
@@ -635,57 +635,34 @@ def fillTemplate(item,
     return fill_template(template, values)
 
 @registry.register_segment("lambda")
-class EvalExpression(AbstractSegment):
+class EvalExpression(AbstractFieldSegment):
     """Evaluate a Python expression on each item in the input stream.
     
     This segment pre-compiles the expression during initialization for efficiency 
     and then applies it to each item during transformation. Expressions are evaluated
     in a restricted environment for security.
     
-    The item is available in expressions as 'item'. If the item is a dictionary,
-    its fields can be accessed directly as variables in the expression.
-    
+    The item is available in expressions as 'item'.     
     """
     
     def __init__(self, 
                  expression: Annotated[str, "The Python expression to evaluate"],
                  field: Annotated[Optional[str], "If provided, extract this field from each item before evaluating"] = "_",
-                 set_as: Annotated[Optional[str], "If provided, append the result to each item under this field name"] = None,
-                 fail_on_error: Annotated[bool, "If True, raises exceptions when evaluation fails. If False, logs errors and returns None"] = True):
+                 set_as: Annotated[Optional[str], "If provided, append the result to each item under this field name"] = None):
         super().__init__()
         self.expression = expression
         self.field = field
         self.set_as = set_as
-        self.fail_on_error = fail_on_error
         
         # Compile the expression into a lambda function
-        self.lambda_function = compileLambda(expression, fail_on_error)
+        self.lambda_function = compileLambda(expression)
     
-    def transform(self, input_iter: Iterable[Any]) -> Iterator[Any]:
-        """Process each item from the input stream."""
-        for item in input_iter:
-            logger.debug(f"Processing input item: {item}")
-            
-            # Extract field value if specified
-            if self.field is not None:
-                value = extract_property(item, self.field)
-                logger.debug(f"Extracted value from field {self.field}: {value}")
-            else:
-                value = item
-                logger.debug(f"Using item directly: {value}")
-            
-            # Evaluate the expression on the value
-            result = self.lambda_function(value)
-            logger.debug(f"Evaluation result: {result}")
-            
-            # Handle the result according to set_as
-            if self.set_as is not None and hasattr(item, '__setitem__'):
-                logger.debug(f"Appending result to field {self.set_as}")
-                item[self.set_as] = result
-                yield item
-            else:
-                logger.debug("Yielding result directly")
-                yield result
+    def process_value(self, value):
+        try:
+            return self.lambda_function(value)
+        except Exception as e:
+            logger.error(f"Error evaluating expression '{self.expression}' on value '{value}': {e}")
+            raise
 
 @registry.register_segment("lambdaFilter")
 class FilterExpression(AbstractSegment):
@@ -702,15 +679,13 @@ class FilterExpression(AbstractSegment):
     
     def __init__(self, 
                  expression: Annotated[str, "The Python expression to evaluate"],
-                 field: Annotated[Optional[str], "If provided, extract this field from each item before evaluating"] = "_",
-                 fail_on_error: Annotated[bool, "If True, raises exceptions when evaluation fails. If False, logs errors and returns None"] = True):
+                 field: Annotated[Optional[str], "If provided, extract this field from each item before evaluating"] = "_"):
         super().__init__()
         self.expression = expression
         self.field = field
-        self.fail_on_error = fail_on_error
         
         # Compile the expression into a lambda function
-        self.lambda_function = compileLambda(expression, fail_on_error)
+        self.lambda_function = compileLambda(expression)
         logger.debug(f"Compiled filter expression: {self.expression}")
     
     def transform(self, input_iter: Iterable[Any]) -> Iterator[Any]: