TalkPipe 0.7.0__tar.gz → 0.7.0a2__tar.gz

{talkpipe-0.7.0 → talkpipe-0.7.0a2}/CHANGELOG.md

@@ -1,5 +1,6 @@
 # Changelog
-## 0.7.0
+
+## In Progress
 ### New Segments and Sources
  - **progressTicks** – Prints out a tick for every n items seen and a new line for every m ticks.  
   Especially useful for debugging and marking progress in logs.
@@ -11,10 +12,6 @@
 ### Improvements
  - Rewrote firstN to use decorator syntax (making it shorter) and to not throw an exception
   when there are fewer than N items.
- - Created a set of protocols for search engines, both full-text and vector DB-based.  
-  These protocols can be used to write wrappers around different search engines and 
-  vector databases, providing a common API that can then be used in different segments
-  and sources.
  - Integrated the Whoosh pure-Python, stand-alone search engine. This is intended for testing,
   debugging, and smaller stand-alone scenarios. It allows a programmer to work on pipelines that 
   use full-text search engines without setting up a full search engine server. It can support
@@ -23,9 +20,6 @@
  - Added SimpleVectorDB, a pure Python, simple vector database. It allows a programmer to build
   and debug pipelines requiring simple vector databases. It supports smaller-scale databases
   that can later be replaced with a separate server.
- - Added a default option to extract_property and propagated the option to several segments.
- - Wrote a conceptual diagram for the library and wrote short descriptions to orient people to 
-  how the library can be used.
 
 ## 0.6.0
 ### New Segments and Sources

{talkpipe-0.7.0 → talkpipe-0.7.0a2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: talkpipe
-Version: 0.7.0
+Version: 0.7.0a2
 Summary: Python internal and external DSL for writing generative AI analytics
 Author-email: Travis Bauer <tlbauer@sandia.gov>
 Maintainer-email: Travis Bauer <tlbauer@sandia.gov>
@@ -56,46 +56,6 @@ TalkPipe is a Python toolkit that makes it easy to create, test, and deploy work
 - **Create AI Agents** - Build agents that can debate topics, evaluate streams of documents, or monitor RSS feeds
 - **Deploy Anywhere** - Run in Jupyter notebooks, as Docker containers, or as standalone Python applications
 
-## Structure
-
-<center><img src="docs/talkpipe_diagram.png" width=700></center>
-
-Talkpipe is structured in five layers: Key Functionality, Pipe, ChatterLang, Applications Components, and Docker Containers.
-
-The Key Functionality layer offers foundational utilities and abstractions that ensure consistency across the entire system. It includes configuration management, as well as standardized wrappers for interacting with LLMs, full-text search engines, and a lightweight vector database. These components provide a unified interface for core capabilities, making it easy to integrate advanced AI and search features throughout your workflows.
-
-The Pipe layer serves as TalkPipe's internal domain-specific language (DSL), enabling you to construct sophisticated workflows directly in Python. By instantiating modular classes and chaining them together using the `|` (pipe) operator, you can seamlessly connect sources, segments, and sinks to process data in a clear, readable, and composable manner. This approach leverages Python's native syntax and type checking, allowing for rapid development, easy debugging, and full IDE support. The Pipe layer is ideal for users who prefer programmatic control and want to integrate TalkPipe pipelines into larger Python applications or scripts.
-
-The ChatterLang layer serves as TalkPipe's external domain-specific language (DSL), enabling you to define workflows using concise, human-readable text scripts. These scripts can be compiled directly in Python, producing callable functions that integrate seamlessly with your codebase. ChatterLang also supports specifying workflows via environment variables or command-line arguments, making it easy to configure and automate pipelines in Docker containers, shell scripts, CI/CD pipelines, and other deployment environments. This flexibility empowers both developers and non-developers to create, share, and modify AI-powered workflows without writing Python code, streamlining experimentation and operationalization.
-
-The Application Components layer contains runnable applications that provide user interfaces and automation tools for working with TalkPipe pipelines and ChatterLang scripts. These components are designed to make it easy to interact with TalkPipe from the command line, web browser, or as part of automated workflows.
-
-### Key Applications
-
-- **chatterlang_server**  
-  Launches an interactive web interface for writing, testing, and running ChatterLang scripts. It provides real-time execution, logging, and documentation lookup.
-
-- **chatterlang_script**  
-  Runs ChatterLang scripts from files or directly from the command line, enabling batch processing and automation.
-
-- **talkpipe_ref**  
-  Generates HTML and text documentation for all available sources and segments, making it easy to discover and understand built-in functionality.
-
-- **talkpipe_endpoint**  
-  Exposes ChatterLang pipelines as REST APIs or web forms, allowing you to deploy workflows as web services or user-facing endpoints.
-
-These applications are entry points for different usage scenarios, from interactive development to production deployment.
-
-The Docker layer provides ready-to-use container images for deploying TalkPipe in different environments. It includes:
-
-- **talkpipe-base**  
-  A minimal container image that contains all dependencies required by TalkPipe, but does not install TalkPipe itself. Use this as a foundation for building your own images, custom workflows, or CI/CD jobs where you want to control the installation of TalkPipe or add additional components.
-
-- **chatterlang_script**  
-  An image that installs TalkPipe on top of the base dependencies and runs a talkpipe script.
-
-These containers exist to simplify deployment, ensure consistent environments, and enable rapid experimentation or production use of TalkPipe and ChatterLang across platforms.
-
 ## Quick Start
 
 Install TalkPipe:

{talkpipe-0.7.0 → talkpipe-0.7.0a2}/README.md

@@ -13,46 +13,6 @@ TalkPipe is a Python toolkit that makes it easy to create, test, and deploy work
 - **Create AI Agents** - Build agents that can debate topics, evaluate streams of documents, or monitor RSS feeds
 - **Deploy Anywhere** - Run in Jupyter notebooks, as Docker containers, or as standalone Python applications
 
-## Structure
-
-<center><img src="docs/talkpipe_diagram.png" width=700></center>
-
-Talkpipe is structured in five layers: Key Functionality, Pipe, ChatterLang, Applications Components, and Docker Containers.
-
-The Key Functionality layer offers foundational utilities and abstractions that ensure consistency across the entire system. It includes configuration management, as well as standardized wrappers for interacting with LLMs, full-text search engines, and a lightweight vector database. These components provide a unified interface for core capabilities, making it easy to integrate advanced AI and search features throughout your workflows.
-
-The Pipe layer serves as TalkPipe's internal domain-specific language (DSL), enabling you to construct sophisticated workflows directly in Python. By instantiating modular classes and chaining them together using the `|` (pipe) operator, you can seamlessly connect sources, segments, and sinks to process data in a clear, readable, and composable manner. This approach leverages Python's native syntax and type checking, allowing for rapid development, easy debugging, and full IDE support. The Pipe layer is ideal for users who prefer programmatic control and want to integrate TalkPipe pipelines into larger Python applications or scripts.
-
-The ChatterLang layer serves as TalkPipe's external domain-specific language (DSL), enabling you to define workflows using concise, human-readable text scripts. These scripts can be compiled directly in Python, producing callable functions that integrate seamlessly with your codebase. ChatterLang also supports specifying workflows via environment variables or command-line arguments, making it easy to configure and automate pipelines in Docker containers, shell scripts, CI/CD pipelines, and other deployment environments. This flexibility empowers both developers and non-developers to create, share, and modify AI-powered workflows without writing Python code, streamlining experimentation and operationalization.
-
-The Application Components layer contains runnable applications that provide user interfaces and automation tools for working with TalkPipe pipelines and ChatterLang scripts. These components are designed to make it easy to interact with TalkPipe from the command line, web browser, or as part of automated workflows.
-
-### Key Applications
-
-- **chatterlang_server**  
-  Launches an interactive web interface for writing, testing, and running ChatterLang scripts. It provides real-time execution, logging, and documentation lookup.
-
-- **chatterlang_script**  
-  Runs ChatterLang scripts from files or directly from the command line, enabling batch processing and automation.
-
-- **talkpipe_ref**  
-  Generates HTML and text documentation for all available sources and segments, making it easy to discover and understand built-in functionality.
-
-- **talkpipe_endpoint**  
-  Exposes ChatterLang pipelines as REST APIs or web forms, allowing you to deploy workflows as web services or user-facing endpoints.
-
-These applications are entry points for different usage scenarios, from interactive development to production deployment.
-
-The Docker layer provides ready-to-use container images for deploying TalkPipe in different environments. It includes:
-
-- **talkpipe-base**  
-  A minimal container image that contains all dependencies required by TalkPipe, but does not install TalkPipe itself. Use this as a foundation for building your own images, custom workflows, or CI/CD jobs where you want to control the installation of TalkPipe or add additional components.
-
-- **chatterlang_script**  
-  An image that installs TalkPipe on top of the base dependencies and runs a talkpipe script.
-
-These containers exist to simplify deployment, ensure consistent environments, and enable rapid experimentation or production use of TalkPipe and ChatterLang across platforms.
-
 ## Quick Start
 
 Install TalkPipe:

{talkpipe-0.7.0 → talkpipe-0.7.0a2}/src/talkpipe/__init__.py

@@ -16,4 +16,4 @@ from talkpipe.operations.matrices import *
 from talkpipe.operations.signatures import *
 from talkpipe.app.apiendpoint import *
 from talkpipe.search.whoosh import *
-from talkpipe.search.simplevectordb import *
+from talkpipe.vectordb.simplevectordb import *

{talkpipe-0.7.0 → talkpipe-0.7.0a2}/src/talkpipe/app/static/unit-docs.html

@@ -451,17 +451,6 @@
   <td class="toc-talkpipe">scale</td>
   <td class="toc-description">Scale each item in the input stream by the multiplier.</td>
 </tr>
-<tr><td class="toc-package" colspan="3"><a href="#talkpipe-search-simplevectordb">talkpipe.search.simplevectordb</a></td></tr>
-<tr>
-  <td class="toc-name"><a href="#talkpipe-search-simplevectordb-add_vector">add_vector</a></td>
-  <td class="toc-talkpipe">addVector</td>
-  <td class="toc-description">Segment to add a vector to the SimpleVectorDB.</td>
-</tr>
-<tr>
-  <td class="toc-name"><a href="#talkpipe-search-simplevectordb-search_vector">search_vector</a></td>
-  <td class="toc-talkpipe">searchVector</td>
-  <td class="toc-description">Segment to search for similar vectors in the SimpleVectorDB.</td>
-</tr>
 <tr><td class="toc-package" colspan="3"><a href="#talkpipe-search-whoosh">talkpipe.search.whoosh</a></td></tr>
 <tr>
   <td class="toc-name"><a href="#talkpipe-search-whoosh-indexWhoosh">indexWhoosh</a></td>
@@ -472,6 +461,17 @@
   <td class="toc-name"><a href="#talkpipe-search-whoosh-searchWhoosh">searchWhoosh</a></td>
   <td class="toc-talkpipe">searchWhoosh</td>
   <td class="toc-description">Search documents using Whoosh full-text indexing.</td>
+</tr>
+<tr><td class="toc-package" colspan="3"><a href="#talkpipe-vectordb-simplevectordb">talkpipe.vectordb.simplevectordb</a></td></tr>
+<tr>
+  <td class="toc-name"><a href="#talkpipe-vectordb-simplevectordb-add_vector">add_vector</a></td>
+  <td class="toc-talkpipe">addVector</td>
+  <td class="toc-description">Segment to add a vector to the SimpleVectorDB.</td>
+</tr>
+<tr>
+  <td class="toc-name"><a href="#talkpipe-vectordb-simplevectordb-search_vector">search_vector</a></td>
+  <td class="toc-talkpipe">searchVector</td>
+  <td class="toc-description">Segment to search for similar vectors in the SimpleVectorDB.</td>
 </tr>
         </tbody>
       </table>
@@ -1319,7 +1319,6 @@ Returns:
   <ul class="param-list">
     <li>template: str</li>
     <li>fail_on_missing: bool = True</li>
-    <li>default: Optional[Any] = &#x27;&#x27;</li>
   </ul>
 </div>
 <div class="class" id="talkpipe-pipe-basic-FilterExpression">
@@ -1539,7 +1538,6 @@ dictionary represents a row in the DataFrame.</pre></div>
     <li>self</li>
     <li>field_list: str = &#x27;_&#x27;</li>
     <li>fail_on_missing: bool = True</li>
-    <li>default: Optional[Any] = None</li>
   </ul>
   <p><strong>Base Classes:</strong> AbstractSegment</p>
 </div>
@@ -1859,8 +1857,42 @@ Raises:
     <li>multiplier: Union[int, float]</li>
   </ul>
 </div>
-<div class="package-header" id="talkpipe-search-simplevectordb">talkpipe.search.simplevectordb</div>
-<div class="function" id="talkpipe-search-simplevectordb-add_vector">
+<div class="package-header" id="talkpipe-search-whoosh">talkpipe.search.whoosh</div>
+<div class="function" id="talkpipe-search-whoosh-indexWhoosh">
+  <h2>Segment Function: indexWhoosh</h2>
+  <p class="talkpipe-name">Chatterlang Name: indexWhoosh</p>
+  <div><pre>Index documents using Whoosh full-text indexing.
+
+Args:
+    fields (list[str]): List of fields to index.
+    index_path (str): Path to the Whoosh index directory.
+    yield_doc (bool): If True, yield each indexed document.  Otherwise yield the original
+    item.</pre></div>
+  <h3>Parameters:</h3>
+  <ul class="param-list">
+    <li>index_path: str</li>
+    <li>field_list: list[str] = [&#x27;_:content&#x27;]</li>
+    <li>yield_doc = False</li>
+  </ul>
+</div>
+<div class="function" id="talkpipe-search-whoosh-searchWhoosh">
+  <h2>Segment Function: searchWhoosh</h2>
+  <p class="talkpipe-name">Chatterlang Name: searchWhoosh</p>
+  <div><pre>Search documents using Whoosh full-text indexing.
+
+Args:
+    index_path (str): Path to the Whoosh index directory.
+    limit (int): Maximum number of results to return for each query.  Defaults to 100.
+    all_results_at_once (bool): If True, yield all results at once. Otherwise, yield one result at a time.</pre></div>
+  <h3>Parameters:</h3>
+  <ul class="param-list">
+    <li>index_path: str</li>
+    <li>limit: int = 100</li>
+    <li>all_results_at_once: bool = False</li>
+  </ul>
+</div>
+<div class="package-header" id="talkpipe-vectordb-simplevectordb">talkpipe.vectordb.simplevectordb</div>
+<div class="function" id="talkpipe-vectordb-simplevectordb-add_vector">
   <h2>Segment Function: add_vector</h2>
   <p class="talkpipe-name">Chatterlang Name: addVector</p>
   <div><pre>Segment to add a vector to the SimpleVectorDB.
@@ -1883,7 +1915,7 @@ Returns:
     <li>overwrite: bool = False</li>
   </ul>
 </div>
-<div class="function" id="talkpipe-search-simplevectordb-search_vector">
+<div class="function" id="talkpipe-vectordb-simplevectordb-search_vector">
   <h2>Segment Function: search_vector</h2>
   <p class="talkpipe-name">Chatterlang Name: searchVector</p>
   <div><pre>Segment to search for similar vectors in the SimpleVectorDB.
@@ -1894,7 +1926,7 @@ Args:
     search_method: Search method ("brute-force", "brute-force-heap", or "k-means").
     path: Optional path to a saved vector database.
 Yields:
-    List of SearchResult objects.</pre></div>
+    List of tuples containing (vector_id, similarity_score, VectorRecord).</pre></div>
   <h3>Parameters:</h3>
   <ul class="param-list">
     <li>path: str</li>
@@ -1903,51 +1935,6 @@ Yields:
     <li>search_metric: str = &#x27;cosine&#x27;</li>
     <li>search_method: str = &#x27;brute-force&#x27;</li>
   </ul>
-</div>
-<div class="package-header" id="talkpipe-search-whoosh">talkpipe.search.whoosh</div>
-<div class="function" id="talkpipe-search-whoosh-indexWhoosh">
-  <h2>Segment Function: indexWhoosh</h2>
-  <p class="talkpipe-name">Chatterlang Name: indexWhoosh</p>
-  <div><pre>Index documents using Whoosh full-text indexing.
-
-Args:
-    items: Iterator of items to index
-    index_path (str): Path to the Whoosh index directory.
-    field_list (list[str]): List of fields to index.
-    yield_doc (bool): If True, yield each indexed document. Otherwise yield the original item.
-    continue_on_error (bool): If True, continue processing other documents when one fails.
-    overwrite (bool): If True, clear existing index before indexing.
-    commit_seconds (int): If > 0, commit changes if it has been this many seconds since the last commit.</pre></div>
-  <h3>Parameters:</h3>
-  <ul class="param-list">
-    <li>index_path: str</li>
-    <li>field_list: list[str] = [&#x27;_:content&#x27;]</li>
-    <li>yield_doc = False</li>
-    <li>continue_on_error = True</li>
-    <li>overwrite = False</li>
-    <li>commit_seconds: int = -1</li>
-  </ul>
-</div>
-<div class="function" id="talkpipe-search-whoosh-searchWhoosh">
-  <h2>Segment Function: searchWhoosh</h2>
-  <p class="talkpipe-name">Chatterlang Name: searchWhoosh</p>
-  <div><pre>Search documents using Whoosh full-text indexing.
-
-Args:
-    queries: Iterator of query strings
-    index_path (str): Path to the Whoosh index directory.
-    limit (int): Maximum number of results to return for each query. Defaults to 100.
-    all_results_at_once (bool): If True, yield all results at once. Otherwise, yield one result at a time.
-    continue_on_error (bool): If True, continue with next query when one fails.
-    reload_seconds (int): If > 0, reload the index if the last search was at least this many seconds ago.</pre></div>
-  <h3>Parameters:</h3>
-  <ul class="param-list">
-    <li>index_path: str</li>
-    <li>limit: int = 100</li>
-    <li>all_results_at_once: bool = False</li>
-    <li>continue_on_error = True</li>
-    <li>reload_seconds: int = 60</li>
-  </ul>
 </div>
   </div>
 </body>

{talkpipe-0.7.0 → talkpipe-0.7.0a2}/src/talkpipe/app/static/unit-docs.txt

@@ -810,7 +810,6 @@ Segment Function: fillTemplate
   Parameters:
     template: str
     fail_on_missing: bool = True
-    default: Optional[Any] = ''
 
 Segment Class: FilterExpression
   Chatterlang Name: lambdaFilter
@@ -1004,7 +1003,6 @@ Segment Class: ToDict
   Parameters:
     field_list: str = '_'
     fail_on_missing: bool = True
-    default: Optional[Any] = None
 
 Segment Class: ToList
   Chatterlang Name: toList
@@ -1285,8 +1283,41 @@ Segment Function: scale
     multiplier: Union[int, float]
 
 
-PACKAGE: talkpipe.search.simplevectordb
----------------------------------------
+PACKAGE: talkpipe.search.whoosh
+-------------------------------
+
+Segment Function: indexWhoosh
+  Chatterlang Name: indexWhoosh
+  Docstring:
+    Index documents using Whoosh full-text indexing.
+    
+    Args:
+        fields (list[str]): List of fields to index.
+        index_path (str): Path to the Whoosh index directory.
+        yield_doc (bool): If True, yield each indexed document.  Otherwise yield the original
+        item.
+  Parameters:
+    index_path: str
+    field_list: list[str] = ['_:content']
+    yield_doc = False
+
+Segment Function: searchWhoosh
+  Chatterlang Name: searchWhoosh
+  Docstring:
+    Search documents using Whoosh full-text indexing.
+    
+    Args:
+        index_path (str): Path to the Whoosh index directory.
+        limit (int): Maximum number of results to return for each query.  Defaults to 100.
+        all_results_at_once (bool): If True, yield all results at once. Otherwise, yield one result at a time.
+  Parameters:
+    index_path: str
+    limit: int = 100
+    all_results_at_once: bool = False
+
+
+PACKAGE: talkpipe.vectordb.simplevectordb
+-----------------------------------------
 
 Segment Function: add_vector
   Chatterlang Name: addVector
@@ -1320,7 +1351,7 @@ Segment Function: search_vector
         search_method: Search method ("brute-force", "brute-force-heap", or "k-means").
         path: Optional path to a saved vector database.
     Yields:
-        List of SearchResult objects.
+        List of tuples containing (vector_id, similarity_score, VectorRecord).
   Parameters:
     path: str
     vector_field = '_'
@@ -1329,47 +1360,3 @@ Segment Function: search_vector
     search_method: str = 'brute-force'
 
 
-PACKAGE: talkpipe.search.whoosh
--------------------------------
-
-Segment Function: indexWhoosh
-  Chatterlang Name: indexWhoosh
-  Docstring:
-    Index documents using Whoosh full-text indexing.
-    
-    Args:
-        items: Iterator of items to index
-        index_path (str): Path to the Whoosh index directory.
-        field_list (list[str]): List of fields to index.
-        yield_doc (bool): If True, yield each indexed document. Otherwise yield the original item.
-        continue_on_error (bool): If True, continue processing other documents when one fails.
-        overwrite (bool): If True, clear existing index before indexing.
-        commit_seconds (int): If > 0, commit changes if it has been this many seconds since the last commit.
-  Parameters:
-    index_path: str
-    field_list: list[str] = ['_:content']
-    yield_doc = False
-    continue_on_error = True
-    overwrite = False
-    commit_seconds: int = -1
-
-Segment Function: searchWhoosh
-  Chatterlang Name: searchWhoosh
-  Docstring:
-    Search documents using Whoosh full-text indexing.
-    
-    Args:
-        queries: Iterator of query strings
-        index_path (str): Path to the Whoosh index directory.
-        limit (int): Maximum number of results to return for each query. Defaults to 100.
-        all_results_at_once (bool): If True, yield all results at once. Otherwise, yield one result at a time.
-        continue_on_error (bool): If True, continue with next query when one fails.
-        reload_seconds (int): If > 0, reload the index if the last search was at least this many seconds ago.
-  Parameters:
-    index_path: str
-    limit: int = 100
-    all_results_at_once: bool = False
-    continue_on_error = True
-    reload_seconds: int = 60
-
-

{talkpipe-0.7.0 → talkpipe-0.7.0a2}/src/talkpipe/pipe/basic.py

@@ -124,7 +124,7 @@ class Cast(AbstractSegment):
 class ToDict(AbstractSegment):
     """Creates a dictionary from the input data."""
 
-    def __init__(self, field_list: str = "_", fail_on_missing: bool = True, default: Optional[Any] = None):
+    def __init__(self, field_list: str = "_", fail_on_missing: bool = True):
         """Convert each item in the input string into a dictionary based on the provided parameter list.
 
         Args:
@@ -392,7 +392,7 @@ class ConfigureLogger(AbstractSegment):
         """
         yield from input_iter
 
-def hash_data(data, algorithm="MD5", field_list="_", use_repr=False, fail_on_missing=True, default=None):
+def hash_data(data, algorithm="MD5", field_list="_", use_repr=False, fail_on_missing=True):
     """Hash a single data item using the specified parameters.
     
     Args:
@@ -407,7 +407,7 @@ def hash_data(data, algorithm="MD5", field_list="_", use_repr=False, fail_on_mis
     """
     hasher = hashlib.new(algorithm)
     for field in field_list:
-        item = extract_property(data, field, fail_on_missing, default=default)
+        item = extract_property(data, field, fail_on_missing)
         if item is None:
             if fail_on_missing:
                 raise ValueError(f"Field {field} value was None")
@@ -459,7 +459,7 @@ class Hash(AbstractSegment):
 
 @registry.register_segment("fillTemplate")
 @field_segment
-def fillTemplate(item, template:str, fail_on_missing:bool=True, default: Optional[Any] = ""):
+def fillTemplate(item, template:str, fail_on_missing:bool=True):
     """Fill a template string with values from the input item.
 
     Args:
@@ -471,7 +471,7 @@ def fillTemplate(item, template:str, fail_on_missing:bool=True, default: Optiona
     """
     assert template is not None
     field_names = extract_template_field_names(template)
-    values = {field_name:extract_property(item, field_name, fail_on_missing=fail_on_missing, default=default) for field_name in field_names}
+    values = {field_name:extract_property(item, field_name, fail_on_missing=fail_on_missing) for field_name in field_names}
     return fill_template(template, values)
 
 @registry.register_segment("lambda")

talkpipe-0.7.0a2/src/talkpipe/search/abstract.py

@@ -0,0 +1,43 @@
+from abc import ABC, abstractmethod
+from typing import List, Dict, Any, Optional
+import uuid
+from talkpipe.util.data_manipulation import extract_property
+from whoosh import index
+from whoosh.fields import Schema, TEXT, ID, STORED
+from whoosh.qparser import MultifieldParser
+import os
+
+class SearchResult:
+    """A single search result."""
+    def __init__(self, doc_id: str, score: float, snippet: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None):
+        self.doc_id = doc_id
+        self.score = score
+        self.snippet = snippet
+        self.metadata = metadata or {}
+
+class AbstractFullTextIndex(ABC):
+    """Abstract base class for a full text index."""
+
+    @abstractmethod
+    def add_document(self, document: dict[str, Any]) -> None:
+        """Add or update a document in the index."""
+        pass
+
+    @abstractmethod
+    def search(self, query: str, limit: int = 10) -> List[SearchResult]:
+        """Search for documents matching the query."""
+        pass
+
+    @abstractmethod
+    def clear(self) -> None:
+        """Remove all documents from the index."""
+        pass
+
+    def __enter__(self):
+        """Support context manager for resource management."""
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        """Support context manager for resource management."""
+        pass
+

talkpipe-0.7.0a2/src/talkpipe/search/whoosh.py

@@ -0,0 +1,127 @@
+from typing import List, Any
+import os
+import uuid
+from whoosh import index
+from whoosh.fields import Schema, TEXT, ID
+from whoosh.qparser import MultifieldParser
+from talkpipe.pipe import segment
+from talkpipe.chatterlang import register_segment
+from talkpipe.util.data_manipulation import toDict
+from talkpipe.util.config import parse_key_value_str
+from .abstract import AbstractFullTextIndex, SearchResult
+
+class WhooshFullTextIndex(AbstractFullTextIndex):
+    def __init__(self, index_path: str, fields: list[str] = None):
+        self.index_path = index_path
+        self.fields = fields
+        if not os.path.exists(index_path):
+            os.makedirs(index_path)
+        if index.exists_in(index_path):
+            self.ix = index.open_dir(index_path)
+            self.schema = self.ix.schema
+            ix_fields = set(self.ix.schema.names())
+            expected_fields = set(self.ix.schema.names())
+            if fields is not None:
+                expected_fields = set(['doc_id'] + fields)
+                if ix_fields != expected_fields:
+                    raise ValueError(f"Index schema fields {ix_fields} do not match expected {expected_fields}")
+                self.fields = fields
+            else:
+                # Use fields from existing index if not provided
+                self.fields = [f for f in self.ix.schema.names() if f != 'doc_id']
+        else:
+            if fields is None:
+                raise ValueError("Fields must be provided when creating a new index.")
+            self.schema = Schema(
+                doc_id=ID(stored=True, unique=True),
+                **{field: TEXT(stored=True) for field in fields}
+            )
+            self.ix = index.create_in(index_path, self.schema)
+        self.writer = None
+
+    def __enter__(self):
+        self.writer = self.ix.writer()
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        if self.writer is not None:
+            self.writer.commit()
+            self.writer = None
+        self.ix.close()
+
+    def add_document(self, document: dict[str, Any]) -> None:
+        doc_id = str(document.get("doc_id") or uuid.uuid4())
+        doc_fields = {field: str(document.get(field, "")) for field in self.fields}
+        if self.writer is not None:
+            self.writer.update_document(doc_id=doc_id, **doc_fields)
+        else:
+            with self.ix.writer() as writer:
+                writer.update_document(doc_id=doc_id, **doc_fields)
+
+    def search(self, query: str, limit: int = 100) -> List[SearchResult]:
+        with self.ix.searcher() as searcher:
+            parser = MultifieldParser(self.fields, schema=self.ix.schema)
+            q = parser.parse(query)
+            results = searcher.search(q, limit=limit)
+            search_results = []
+            for hit in results:
+                metadata = {field: hit.get(field) for field in self.fields}
+                search_results.append(
+                    SearchResult(
+                        doc_id=hit['doc_id'],
+                        score=hit.score,
+                        snippet=None,
+                        metadata=metadata
+                    )
+                )
+            return search_results
+
+    def clear(self) -> None:
+        if self.writer is not None:
+            self.writer.cancel()
+            self.writer = None
+        self.ix.close()
+        for fname in os.listdir(self.index_path):
+            fpath = os.path.join(self.index_path, fname)
+            if os.path.isfile(fpath):
+                os.remove(fpath)
+        self.ix = index.create_in(self.index_path, self.schema)
+
+@register_segment("indexWhoosh")
+@segment()
+def indexWhoosh(items, index_path: str, field_list: list[str] = ["_:content"], yield_doc=False):
+    """Index documents using Whoosh full-text indexing.
+
+    Args:
+        fields (list[str]): List of fields to index.
+        index_path (str): Path to the Whoosh index directory.
+        yield_doc (bool): If True, yield each indexed document.  Otherwise yield the original
+        item.
+    """
+    field_list_dict = parse_key_value_str(field_list)
+    with WhooshFullTextIndex(index_path, list(field_list_dict.values())) as idx:
+        for item in items:
+            d = toDict(item, field_list, fail_on_missing=False)
+            d['doc_id'] = str(d.get('doc_id', uuid.uuid4()))
+            d_str = {k:str(v) for k, v in d.items()}
+            idx.add_document(d_str)
+            yield d if yield_doc else item
+
+@register_segment("searchWhoosh")
+@segment()
+def searchWhoosh(queries, index_path: str, limit: int = 100, all_results_at_once: bool = False):
+    """Search documents using Whoosh full-text indexing.
+
+    Args:
+        index_path (str): Path to the Whoosh index directory.
+        limit (int): Maximum number of results to return for each query.  Defaults to 100.
+        all_results_at_once (bool): If True, yield all results at once. Otherwise, yield one result at a time.
+    """
+    with WhooshFullTextIndex(index_path) as idx:
+        for query in queries:
+            results = idx.search(query, limit=limit)
+            if all_results_at_once:
+                yield results
+            else:
+                for result in results:
+                    yield result

{talkpipe-0.7.0 → talkpipe-0.7.0a2}/src/talkpipe/util/data_manipulation.py

@@ -115,7 +115,7 @@ def get_all_attributes(obj: Any, skip_packages: tuple = ('pydantic',), visited:
     return attributes
 
 
-def extract_property(data: Any, prop_list: str, fail_on_missing=False, default=None) -> Any:
+def extract_property(data: Any, prop_list: str, fail_on_missing=False) -> Any:
     """Extract a property from a nested data structure using dot notation.
 
     Args:
@@ -142,7 +142,7 @@ def extract_property(data: Any, prop_list: str, fail_on_missing=False, default=N
             if fail_on_missing:
                 raise AttributeError(f"Property '{prop_name}' not found in the input data of type '{type(data)}'")
             else:
-                return default
+                return None
     return of_interest