TalkPipe 0.7.0a2__tar.gz → 0.7.1__tar.gz

{talkpipe-0.7.0a2 → talkpipe-0.7.1}/.github/ISSUE_TEMPLATE/release-checklist.md

@@ -11,10 +11,13 @@ assignees: ''
 - [ ] Commit all changes
 - [ ] Ensure unit tests pass in live environment
 - [ ] Merge into master
-- [ ] Build docker image
+- [ ] Build docker images and make sure runnable unit tests pass
+        - docker build -f base-image/Dockerfile . -t talkpipe:base
+        - docker build . -t talkpipe:experimental
 - [ ] Tag repository
 - [ ] Build whl files
 - [ ] Install whl file in new environment and test a script in chatterlang_server
+- [ ] Run examples in the tutorials
 - [ ] Push to pypi
 - [ ] Push code to repo
 - [ ] Push tags to repo

{talkpipe-0.7.0a2 → talkpipe-0.7.1}/CHANGELOG.md

@@ -1,6 +1,21 @@
 # Changelog
+## 0.7.1
+### New Segments and Sources
+ - Added **formatItem** to produce a human-readable text representation of various properties of an item, and added relevant methods to data_manipulation.
+ - Added **copy** and **deepcopy** segments for making copies of data in a pipeline. This is important in situations where you 
+   don't want the original input data to be modified during processing.
+
+### Improvements
+ - Added an "examples" directory with three tutorials, each of which builds on the previous one and contains multiple steps.
+ - Updates to **apiendpoint**:
+  - Changed the default port.
+  - Fixed a bug causing it to be unable to load a script from a file.
+ - Added metric and method options to vector_search in the SimpleVectorDB class and propagated those through to searchVector.
+ - Updated **progressTicks** so it writes to stderr rather than stdout.
+ - Updated **searchWhoosh** so the user can specify fields to use as the query and whether to attach results or pass them all along.
+
 
-## In Progress
+## 0.7.0
 ### New Segments and Sources
  - **progressTicks** – Prints out a tick for every n items seen and a new line for every m ticks.  
   Especially useful for debugging and marking progress in logs.
@@ -12,6 +27,10 @@
 ### Improvements
  - Rewrote firstN to use decorator syntax (making it shorter) and to not throw an exception
   when there are fewer than N items.
+ - Created a set of protocols for search engines, both full-text and vector DB-based.  
+  These protocols can be used to write wrappers around different search engines and 
+  vector databases, providing a common API that can then be used in different segments
+  and sources.
  - Integrated the Whoosh pure-Python, stand-alone search engine. This is intended for testing,
   debugging, and smaller stand-alone scenarios. It allows a programmer to work on pipelines that 
   use full-text search engines without setting up a full search engine server. It can support
@@ -20,6 +39,9 @@
  - Added SimpleVectorDB, a pure Python, simple vector database. It allows a programmer to build
   and debug pipelines requiring simple vector databases. It supports smaller-scale databases
   that can later be replaced with a separate server.
+ - Added a default option to extract_property and propagated the option to several segments.
+ - Wrote a conceptual diagram for the library and wrote short descriptions to orient people to 
+  how the library can be used.
 
 ## 0.6.0
 ### New Segments and Sources

{talkpipe-0.7.0a2 → talkpipe-0.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: talkpipe
-Version: 0.7.0a2
+Version: 0.7.1
 Summary: Python internal and external DSL for writing generative AI analytics
 Author-email: Travis Bauer <tlbauer@sandia.gov>
 Maintainer-email: Travis Bauer <tlbauer@sandia.gov>
@@ -24,7 +24,8 @@ Requires-Dist: python-docx
 Requires-Dist: pandas
 Requires-Dist: feedparser
 Requires-Dist: readability-lxml
-Requires-Dist: lxml[html_clean]
+Requires-Dist: lxml
+Requires-Dist: lxml_html_clean
 Requires-Dist: openai
 Requires-Dist: fastapi[standard]
 Requires-Dist: ipywidgets
@@ -56,6 +57,46 @@ TalkPipe is a Python toolkit that makes it easy to create, test, and deploy work
 - **Create AI Agents** - Build agents that can debate topics, evaluate streams of documents, or monitor RSS feeds
 - **Deploy Anywhere** - Run in Jupyter notebooks, as Docker containers, or as standalone Python applications
 
+## Structure
+
+<center><img src="docs/talkpipe_diagram.png" width=700></center>
+
+Talkpipe is structured in five layers: Key Functionality, Pipe, ChatterLang, Applications Components, and Docker Containers.
+
+The Key Functionality layer offers foundational utilities and abstractions that ensure consistency across the entire system. It includes configuration management, as well as standardized wrappers for interacting with LLMs, full-text search engines, and a lightweight vector database. These components provide a unified interface for core capabilities, making it easy to integrate advanced AI and search features throughout your workflows.
+
+The Pipe layer serves as TalkPipe's internal domain-specific language (DSL), enabling you to construct sophisticated workflows directly in Python. By instantiating modular classes and chaining them together using the `|` (pipe) operator, you can seamlessly connect sources, segments, and sinks to process data in a clear, readable, and composable manner. This approach leverages Python's native syntax and type checking, allowing for rapid development, easy debugging, and full IDE support. The Pipe layer is ideal for users who prefer programmatic control and want to integrate TalkPipe pipelines into larger Python applications or scripts.
+
+The ChatterLang layer serves as TalkPipe's external domain-specific language (DSL), enabling you to define workflows using concise, human-readable text scripts. These scripts can be compiled directly in Python, producing callable functions that integrate seamlessly with your codebase. ChatterLang also supports specifying workflows via environment variables or command-line arguments, making it easy to configure and automate pipelines in Docker containers, shell scripts, CI/CD pipelines, and other deployment environments. This flexibility empowers both developers and non-developers to create, share, and modify AI-powered workflows without writing Python code, streamlining experimentation and operationalization.
+
+The Application Components layer contains runnable applications that provide user interfaces and automation tools for working with TalkPipe pipelines and ChatterLang scripts. These components are designed to make it easy to interact with TalkPipe from the command line, web browser, or as part of automated workflows.
+
+### Key Applications
+
+- **chatterlang_server**  
+  Launches an interactive web interface for writing, testing, and running ChatterLang scripts. It provides real-time execution, logging, and documentation lookup.
+
+- **chatterlang_script**  
+  Runs ChatterLang scripts from files or directly from the command line, enabling batch processing and automation.
+
+- **talkpipe_ref**  
+  Generates HTML and text documentation for all available sources and segments, making it easy to discover and understand built-in functionality.
+
+- **talkpipe_endpoint**  
+  Exposes ChatterLang pipelines as REST APIs or web forms, allowing you to deploy workflows as web services or user-facing endpoints.
+
+These applications are entry points for different usage scenarios, from interactive development to production deployment.
+
+The Docker layer provides ready-to-use container images for deploying TalkPipe in different environments. It includes:
+
+- **talkpipe-base**  
+  A minimal container image that contains all dependencies required by TalkPipe, but does not install TalkPipe itself. Use this as a foundation for building your own images, custom workflows, or CI/CD jobs where you want to control the installation of TalkPipe or add additional components.
+
+- **chatterlang_script**  
+  An image that installs TalkPipe on top of the base dependencies and runs a talkpipe script.
+
+These containers exist to simplify deployment, ensure consistent environments, and enable rapid experimentation or production use of TalkPipe and ChatterLang across platforms.
+
 ## Quick Start
 
 Install TalkPipe:

{talkpipe-0.7.0a2 → talkpipe-0.7.1}/README.md

@@ -13,6 +13,46 @@ TalkPipe is a Python toolkit that makes it easy to create, test, and deploy work
 - **Create AI Agents** - Build agents that can debate topics, evaluate streams of documents, or monitor RSS feeds
 - **Deploy Anywhere** - Run in Jupyter notebooks, as Docker containers, or as standalone Python applications
 
+## Structure
+
+<center><img src="docs/talkpipe_diagram.png" width=700></center>
+
+Talkpipe is structured in five layers: Key Functionality, Pipe, ChatterLang, Applications Components, and Docker Containers.
+
+The Key Functionality layer offers foundational utilities and abstractions that ensure consistency across the entire system. It includes configuration management, as well as standardized wrappers for interacting with LLMs, full-text search engines, and a lightweight vector database. These components provide a unified interface for core capabilities, making it easy to integrate advanced AI and search features throughout your workflows.
+
+The Pipe layer serves as TalkPipe's internal domain-specific language (DSL), enabling you to construct sophisticated workflows directly in Python. By instantiating modular classes and chaining them together using the `|` (pipe) operator, you can seamlessly connect sources, segments, and sinks to process data in a clear, readable, and composable manner. This approach leverages Python's native syntax and type checking, allowing for rapid development, easy debugging, and full IDE support. The Pipe layer is ideal for users who prefer programmatic control and want to integrate TalkPipe pipelines into larger Python applications or scripts.
+
+The ChatterLang layer serves as TalkPipe's external domain-specific language (DSL), enabling you to define workflows using concise, human-readable text scripts. These scripts can be compiled directly in Python, producing callable functions that integrate seamlessly with your codebase. ChatterLang also supports specifying workflows via environment variables or command-line arguments, making it easy to configure and automate pipelines in Docker containers, shell scripts, CI/CD pipelines, and other deployment environments. This flexibility empowers both developers and non-developers to create, share, and modify AI-powered workflows without writing Python code, streamlining experimentation and operationalization.
+
+The Application Components layer contains runnable applications that provide user interfaces and automation tools for working with TalkPipe pipelines and ChatterLang scripts. These components are designed to make it easy to interact with TalkPipe from the command line, web browser, or as part of automated workflows.
+
+### Key Applications
+
+- **chatterlang_server**  
+  Launches an interactive web interface for writing, testing, and running ChatterLang scripts. It provides real-time execution, logging, and documentation lookup.
+
+- **chatterlang_script**  
+  Runs ChatterLang scripts from files or directly from the command line, enabling batch processing and automation.
+
+- **talkpipe_ref**  
+  Generates HTML and text documentation for all available sources and segments, making it easy to discover and understand built-in functionality.
+
+- **talkpipe_endpoint**  
+  Exposes ChatterLang pipelines as REST APIs or web forms, allowing you to deploy workflows as web services or user-facing endpoints.
+
+These applications are entry points for different usage scenarios, from interactive development to production deployment.
+
+The Docker layer provides ready-to-use container images for deploying TalkPipe in different environments. It includes:
+
+- **talkpipe-base**  
+  A minimal container image that contains all dependencies required by TalkPipe, but does not install TalkPipe itself. Use this as a foundation for building your own images, custom workflows, or CI/CD jobs where you want to control the installation of TalkPipe or add additional components.
+
+- **chatterlang_script**  
+  An image that installs TalkPipe on top of the base dependencies and runs a talkpipe script.
+
+These containers exist to simplify deployment, ensure consistent environments, and enable rapid experimentation or production use of TalkPipe and ChatterLang across platforms.
+
 ## Quick Start
 
 Install TalkPipe:

talkpipe-0.7.1/docs/.gitignore

@@ -0,0 +1,3 @@
+*.swp
+*.aux
+*~

talkpipe-0.7.1/docs/make_structure_diagram.sh

@@ -0,0 +1,3 @@
+pdflatex structure.tex
+convert -density 300 structure.pdf -quality 100 talkpipe_diagram.png
+rm structure.pdf

talkpipe-0.7.1/docs/structure.tex

@@ -0,0 +1,59 @@
+\documentclass[tikz]{standalone}
+\usepackage{tikz}
+\usetikzlibrary{positioning, shapes, fit, backgrounds, calc, arrows.meta}
+
+% Declare the layers
+\pgfdeclarelayer{background}
+\pgfdeclarelayer{docker}
+\pgfsetlayers{background,docker,main}
+
+\begin{document}
+\begin{tikzpicture}[
+  box/.style={rectangle, draw, rounded corners, minimum width=6cm, minimum height=1cm, align=center, fill=white},
+  keylayer/.style={draw=blue!60, fill=blue!10, thick, rounded corners, inner sep=10pt},
+  dockerlayer/.style={draw=gray!80, fill=gray!20, thick, rounded corners, inner sep=15pt},
+  arrow/.style={->, thick, blue!60, >=stealth},
+  font=\sffamily
+]
+% Key Functionality nodes
+\node[box] (flow)                             {FLOW CONTROL\\\scriptsize (e.g. filtering, expressions, conversions)};
+\node[box, below=0cm of flow] (data)          {DATA MANIPULATION\\\scriptsize (e.g. download, html cleaning)};
+\node[box, below=0cm of data] (llm)           {LLM PRIMITIVES\\\scriptsize (e.g. chat, scoring, embedding)};
+\node[box, below=0cm of llm] (store)          {DATA STORAGE\\\scriptsize (e.g. full text indexing, vector database)};
+\node[box, below=0cm of store] (utils)        {UTILS\\\scriptsize (e.g. configuration)};
+
+% Pipe - increased spacing to accommodate label
+\node[box, above=1.2cm of flow] (pipe)        {PIPE\\\scriptsize Internal DSL for composing AI workflows};
+
+% ChatterLang
+\node[box, above=0.5cm of pipe] (chatter)     {CHATTERLANG\\\scriptsize External DSL for composing AI workflows};
+
+% Application Layer
+\node[box, above=0.5cm of chatter] (app) {
+  \textbf{APPLICATION COMPONENTS}\\
+  \scriptsize Command-line Apps \quad API Endpoints \quad Exploratory Web UIs \quad Prototypes
+};
+
+% Docker container background around all (bottom layer)
+\begin{pgfonlayer}{background}
+  \node[dockerlayer, fit=(app)(utils)] (docker-bg) {};
+\end{pgfonlayer}
+
+% Key Functionality background box (middle layer)
+\begin{pgfonlayer}{docker}
+  \node[keylayer, fit=(flow)(data)(llm)(store)(utils)] (key-bg) {};
+\end{pgfonlayer}
+
+% Key Functionality label (main layer - centered at top)
+\node[at=(key-bg.north), anchor=south, yshift=-2pt, fill=blue!10, inner sep=3pt, draw=blue!60, rounded corners=3pt] (kfbox) {\textbf{KEY FUNCTIONALITY}};
+
+% Docker label (main layer - centered at top)
+\node[at=(docker-bg.north), anchor=south, yshift=-2pt, fill=gray!20, inner sep=3pt, draw=gray!80, rounded corners=3pt] {\textbf{DOCKER CONTAINER}};
+
+% Arrows connecting the components
+\draw[arrow] (kfbox.north) -- (pipe.south);
+\draw[arrow] (pipe.north) -- (chatter.south);
+\draw[arrow] (chatter.north) -- (app.south);
+
+\end{tikzpicture}
+\end{document}

talkpipe-0.7.1/examples/.gitignore

@@ -0,0 +1,2 @@
+full_text_index
+vector_index