aia 0.5.17 → 0.8.0

data/README.md

@@ -1,20 +1,33 @@
-# AI Assistant (AIA)
+3# AI Assistant (AIA)
 
-`aia` is a command-line utility that facilitates interaction with AI models. It automates the management of pre-compositional prompts and executes generative AI (Gen-AI) commands on those prompts.
+**The prompt is the code!**
 
-It leverages the `prompt_manager` gem to manage prompts for the `mods` and `sgpt` CLI utilities. It utilizes "ripgrep" for searching for prompt files.  It uses `fzf` for prompt selection based on a search term and fuzzy matching.
+```plain
+     ,      ,                 AIA is a command-line utility that facilitates
+     (\____/) AI Assistant    interaction with AI models. It automates the
+      (_oo_)   Fancy LLM      management of pre-compositional prompts and
+        (O)     is Online     executes generative AI (Gen-AI) commands on those
+      __||__    \)            prompts. AIA includes enhanced feathres such as
+    [/______\]  /               * embedded directives * shell integration
+   / \__AI__/ \/                * embedded Ruby       * history management
+  /    /__\                     * interactive chat    * prompt workflows
+ (\   /____\
+```
 
-**Most Recent Change**: Refer to the [Changelog](CHANGELOG.md)
+AIA leverages the [prompt_manager gem](https://github.com/madbomber/prompt_manager) to manage prompts. It utilizes the [CLI tool fzf](https://github.com/junegunn/fzf) for prompt selection.
 
-> v0.5.17
-> - replaced gem semver with versionaire
->
-> v0.5.16
-> - fixed bugs with the prompt pipeline
-> - Added new backend "client" which is an `aia` internal client to the OpenAI API that allows both text-to-speech and speech-to-text
-> - Added --image_size and --image_quality to support image generation with the dall-e-2 and dall-e-3 models using the new internal `aia` OpenAI client.
->
+**Most Recent Change**: Refer to the [Changelog](CHANGELOG.md)
 
+**Notable Recent Changes:**
+- **Directive Processing in Chat and Prompts:** You can now use directives in chat sessions and prompt files with the syntax: `//command args`. Supported directives include:
+  - `shell`/`sh`: Execute shell commands
+  - `ruby`/`rb`: Execute Ruby code
+  - `config`/`cfg`: Display or update configuration
+  - `include`/`inc`: Include file content
+  - `next`: Specify the next prompt ID in a sequence
+  - `pipeline`: Specify a pipeline of prompt IDs to process
+  - `clear`: Clear the context (handy in a chat session)
+  - `help`: Show available directives
 
 
 <!-- Tocer[start]: Auto-generated, don't remove. -->
@@ -22,42 +35,46 @@ It leverages the `prompt_manager` gem to manage prompts for the `mods` and `sgpt
 ## Table of Contents
 
   - [Installation](#installation)
+  - [What is a Prompt ID?](#what-is-a-prompt-id)
+  - [Embedded Parameters as Placeholders](#embedded-parameters-as-placeholders)
   - [Usage](#usage)
-  - [Configuration Using Envars](#configuration-using-envars)
+  - [Configuration Options](#configuration-options)
+    - [Configuration Flexibility](#configuration-flexibility)
+    - [Expandable Configuration](#expandable-configuration)
   - [Shell Integration inside of a Prompt](#shell-integration-inside-of-a-prompt)
-      - [Access to System Environment Variables](#access-to-system-environment-variables)
       - [Dynamic Shell Commands](#dynamic-shell-commands)
+      - [Shell Command Safety](#shell-command-safety)
       - [Chat Session Use](#chat-session-use)
   - [*E*mbedded *R*u*B*y (ERB)](#embedded-ruby-erb)
-    - [Chat Session Behavior](#chat-session-behavior)
   - [Prompt Directives](#prompt-directives)
     - [Parameter and Shell Substitution in Directives](#parameter-and-shell-substitution-in-directives)
-    - [`aia` Specific Directive Commands](#aia-specific-directive-commands)
+    - [Directive Syntax](#directive-syntax)
+    - [AIA Specific Directive Commands](#aia-specific-directive-commands)
       - [//config](#config)
       - [//include](#include)
       - [//ruby](#ruby)
       - [//shell](#shell)
-    - [Backend Directive Commands](#backend-directive-commands)
+      - [//next](#next)
+      - [//pipeline](#pipeline)
     - [Using Directives in Chat Sessions](#using-directives-in-chat-sessions)
   - [Prompt Sequences](#prompt-sequences)
     - [--next](#--next)
     - [--pipeline](#--pipeline)
     - [Best Practices ??](#best-practices-)
-    - [Example pipline](#example-pipline)
+    - [Example pipeline](#example-pipeline)
   - [All About ROLES](#all-about-roles)
-    - [The --roles_dir (AIA_ROLES_DIR)](#the---roles_dir-aia_roles_dir)
+    - [The --roles_prefix (AIA_ROLES_PREFIX)](#the---roles_prefix-aia_roles_prefix)
     - [The --role Option](#the---role-option)
     - [Other Ways to Insert Roles into Prompts](#other-ways-to-insert-roles-into-prompts)
   - [External CLI Tools Used](#external-cli-tools-used)
-    - [Optional External CLI-tools](#optional-external-cli-tools)
-      - [Backend Processor `llm`](#backend-processor-llm)
-      - [Backend Processor `sgpt`](#backend-processor-sgpt)
-      - [Occassionally Useful Tool `plz`](#occassionally-useful-tool-plz)
   - [Shell Completion](#shell-completion)
   - [My Most Powerful Prompt](#my-most-powerful-prompt)
   - [My Configuration](#my-configuration)
+  - [Executable Prompts](#executable-prompts)
   - [Development](#development)
   - [Contributing](#contributing)
+  - [History of Development](#history-of-development)
+  - [Roadmap](#roadmap)
   - [License](#license)
 
 <!-- Tocer[finish]: Auto-generated, don't remove. -->
@@ -69,137 +86,251 @@ Install the gem by executing:
 
     gem install aia
 
-
 Install the command-line utilities by executing:
 
-    brew install mods fzf ripgrep
+    brew install fzf
 
 You will also need to establish a directory in your file system where your prompt text files, last used parameters and usage log files are kept.
 
-Setup a system environment variable (envar) named "AIA_PROMPTS_DIR" that points to your prompts directory.  The default is in your HOME directory named ".prompts". The envar "AIA_ROLES_DIR" points to your role directory where you have prompts that define the different roles you want the LLM to assume when it is doing its work.  The default roles directory is inside the prompts directory.  Its name is "roles".
+Setup a system environment variable (envar) named "AIA_PROMPTS_DIR" that points to your prompts directory.  The default is in your HOME directory named ".prompts". The envar "AIA_ROLES_PREFIX" points to your role prefix where you have prompts that define the different roles you want the LLM to assume when it is doing its work.  The default roles prefix is "roles".
 
 You may also want to install the completion script for your shell.  To get a copy of the completion script do:
 
-`aia --completion bash`
+```bash
+aia --completion bash
+```
 
 `fish` and `zsh` are also available.
 
+## What is a Prompt ID?
 
-## Usage
+A prompt ID is the basename of a text file (extension *.txt) located in a prompts directory. The prompts directory is specified by the environment variable "AIA_PROMPTS_DIR". If this variable is not set, the default is in your HOME directory named ".prompts".  It can also be set on the command line with the `--prompts-dir` option.
+
+This file contains the context and instructions for the LLM to follow. The prompt ID is what you use as an option on the command line to specify which prompt text file to use. Prompt files can have comments, parameters, directives and ERB blocks along with the instruction text to feed to the LLM. It can also have shell commands and use system environment variables.  Consider the following example:
+
+```plaintext
+#!/usr/bin/env aia run
+# ~/.prompts/example.txt
+# Desc: Be an example prompt with all? the bells and whistles
+
+# Set the configuration for this prompt
+
+//config model = gpt-4
+//config temperature = 0.7
+//config shell = true
+//config erb = true
+//config out_file = path/to/output.md
+
+# Add some file content to the context/instructions
+
+//include path/to/file
+//shell cat path/to/file
+$(cat path/to/file)
+
+# Setup some workflows
 
-The usage report obtained using either `-h` or `--help` is implemented as a standard `man` page.  You can use both `--help --verbose` of `-h -v` together to get not only the `aia` man page but also the usage report from the `backend` LLM processing tool.
+//next next_prompt_id
+//pipeline prompt_id_1, prompt_ie_2, prompt_id_3
 
-```shell
-$ aia --help
+# Execute some Ruby code
+
+//ruby require 'some_library' # inserts into the context/instructions
+<% some_ruby_things # not inserted into the context %>
+<%= some_other_ruby_things # that are part of the context/instructions %>
+
+Tell me how to do something for a $(uname -s) platform that would rename all
+of the files in the directory $MY_DIRECTORY to have a prefix of for its filename
+that is [PREFIX] and a ${SUFFIX}
+
+# directives, ERB blocks and other junk can be used
+# anywhere in the file mixing dynamic context/instructions with
+# the static stuff.
+
+__END__
+
+Block comments that are not part of the context or instructions to
+the LLM.  Stuff that is just here ofr documentation.
 ```
 
-## Configuration Using Envars
+That is just about everything including the kitchen sink that a pre-compositional parameterized prompt file can have.  It can be an executable with a she-bang line and a special system prompt name `run` as shown in the example.  It has line comments that use the `#` symbol. It had end of file block comments that appear after the "__END__" line.  It has directive command that begin with the double slash `//` - an homage to IBM JCL.  It has shell variables in both forms.  It has shell commands. It has parameters that default to a regex that uses square brackets and all uppercase characeters to define the parameter name whose value is to be given in a Q&A session before the prompt is sent to the LLM for processing.
 
-The `aia` configuration defaults can be over-ridden by system environment variables *(envars)* with the prefix "AIA_" followed by the config item name also in uppercase. All configuration items can be over-ridden in this way by an envar.  The following table show a few examples.
+AIA has the ability to define a workflow of prompt IDs with either the //next or //pipeline directives.
 
-| Config Item   | Default Value | envar key |
-| ------------- | ------------- | --------- |
-| backend       | mods          | AIA_BACKEND |
-| config_file   | nil           | AIA_CONFIG_FILE |
-| debug         | false         | AIA_DEBUG |
-| edit          | false         | AIA_EDIT |
-| extra         | ''            | AIA_EXTRA |
-| fuzzy         | false         | AIA_FUZZY |
-| log_file      | ~/.prompts/_prompts.log | AIA_LOG_FILE |
-| markdown      | true          | AIA_MARKDOWN |
-| model         | gpt-4-1106-preview | AIA_MODEL |
-| out_file      | STDOUT        | AIA_OUT_FILE |
-| prompts_dir   | ~/.prompts    | AIA_PROMPTS_DIR |
-| speech_model. | tts-1         | AIA_SPEECH_MODEL |
-| verbose       | FALSE         | AIA_VERBOSE |
-| voice         | alloy         | AIA_VOICE |
+You could say that instead of the prompt being part of a program, a program can be part of the prompt. **The prompt is the code!**
 
+By using ERB you can make parts of the context/instructions conditional. You can also use ERB to make parts of the context/instructions dynamic for example to pull information from a database or an API.
 
+## Embedded Parameters as Placeholders
 
-See the `@options` hash in the `cli.rb` file for a complete list.  There are some config items that do not necessarily make sense for use as an envar over-ride.  For example if you set `export AIA_DUMP_FILE=config.yaml` then `aia` would dump the current configuration config.yaml and exit every time it is ran until you finally `unset AIA_DUMP_FILE`
+In the example prompt text file above I used the default regex to define parameters as all upper case characters plus space, underscore and the vertical pipe enclosed within square brackets. Since the time that I original starting writing AIA I've seen more developers use double curly braces to define parameters. AIA allows you to specify your own regex as a string. If you want the curly brackets use the `--regex` option on the command line like this:
 
-In addition to these config items for `aia` the optional command line parameters for the backend prompt processing utilities (mods and sgpt) can also be set using envars with the "AIA_" prefix.  For example "export AIA_TOPP=1.0" will set the "--topp 1.0" command line option for the `mods` utility when its used as the backend processor.
+`--regex '(?-mix:({{[a-zA-Z _|]+}}))'`
 
-## Shell Integration inside of a Prompt
 
-Using the option `--shell` enables `aia` to access your terminal's shell environment from inside the prompt text.
+## Usage
 
-#### Access to System Environment Variables
+The usage report is obtained with either `-h` or `--help` options.
+
+```bash
+aia --help
+```
+
+## Configuration Options
+
+The following table provides a comprehensive list of configuration options, their default values, and the associated environment variables:
+
+| Option                  | Default Value                   | Environment Variable      |
+|-------------------------|---------------------------------|---------------------------|
+| out_file                | temp.md                         | AIA_OUT_FILE              |
+| log_file                | ~/.prompts/_prompts.log         | AIA_LOG_FILE              |
+| prompts_dir             | ~/.prompts                      | AIA_PROMPTS_DIR           |
+| roles_prefix            | roles                           | AIA_ROLES_PREFIX          |
+| model                   | gpt-4o-mini                     | AIA_MODEL                 |
+| speech_model            | tts-1                           | AIA_SPEECH_MODEL          |
+| transcription_model     | whisper-1                       | AIA_TRANSCRIPTION_MODEL   |
+| verbose                 | false                           | AIA_VERBOSE               |
+| markdown                | true                            | AIA_MARKDOWN              |
+| shell                   | false                           | AIA_SHELL                 |
+| erb                     | false                           | AIA_ERB                   |
+| chat                    | false                           | AIA_CHAT                  |
+| clear                   | false                           | AIA_CLEAR                 |
+| terse                   | false                           | AIA_TERSE                 |
+| debug                   | false                           | AIA_DEBUG                 |
+| fuzzy                   | false                           | AIA_FUZZY                 |
+| speak                   | false                           | AIA_SPEAK                 |
+| append                  | false                           | AIA_APPEND                |
+| temperature             | 0.7                             | AIA_TEMPERATURE           |
+| max_tokens              | 2048                            | AIA_MAX_TOKENS            |
+| top_p                   | 1.0                             | AIA_TOP_P                 |
+| frequency_penalty       | 0.0                             | AIA_FREQUENCY_PENALTY     |
+| presence_penalty        | 0.0                             | AIA_PRESENCE_PENALTY      |
+| image_size              | 1024x1024                       | AIA_IMAGE_SIZE            |
+| image_quality           | standard                        | AIA_IMAGE_QUALITY         |
+| image_style             | vivid                           | AIA_IMAGE_STYLE           |
+| embedding_model         | text-embedding-ada-002          | AIA_EMBEDDING_MODEL       |
+| speak_command           | afplay                          | AIA_SPEAK_COMMAND         |
+| require_libs            | []                              | AIA_REQUIRE_LIBS          |
+| regex                   | '(?-mix:(\\[[A-Z _|]+\\]))'     | AIA_REGEX                 |
+
+These options can be configured via command-line arguments, environment variables, or configuration files.
+
+### Configuration Flexibility
+
+AIA determines configuration settings using the following order of precedence:
+
+1. Embedded config directives
+2. Command-line arguments
+3. Environment variables
+4. Configuration files
+5. Default values
+
+For example, let's consider the `model` option. Suppose the following conditions:
+
+- Default value is "gpt-4o-mini"
+- No entry in the config file
+- No environment variable value for `AIA_MODEL`
+- No command-line argument provided for `--model`
+- No embedded directive like `//config model = some-fancy-llm`
+
+In this scenario, the model used will be "gpt-4o-mini". However, you can override this default by setting the model at any level of the precedence order. Additionally, you can dynamically ask for user input by incorporating an embedded directive with a placeholder parameter, such as `//config model = [PROCESS_WITH_MODEL]`. When processing the prompt, AIA will prompt you to input a value for `[PROCESS_WITH_MODEL]`.
+
+If you do not like the default regex used to identify parameters within the prompt text, don't worry there is a way to configure it using the `--regex` option.
+
+### Expandable Configuration
+
+The configuration options are expandable through a config file, allowing you to add custom entries. For example, you can define a custom configuration item like "xyzzy" in your config file. This value can then be accessed in your prompts using `AIA.config.xyzzy` within a `//ruby` directive or an ERB block, enabling dynamic prompt generation based on your custom configurations.
 
-`aia` can replace any system environment variable (envar) references in the prompt text with the value of the envar.  Patterns like $USER and ${USER} in the prompt will be replaced with that envar's value - the name of the user's account.  Any envar can be used.
+## Shell Integration inside of a Prompt
+
+Using the option `--shell` enables AIA to access your terminal's shell environment from inside the prompt text.
 
 #### Dynamic Shell Commands
 
-Dynamic content can be inserted into the prompt using the pattern $(shell command) where the output of the shell command will replace the $(...) pattern.
+Dynamic content can be inserted into the prompt using the pattern $(shell command) where the output of the shell command will replace the $(...) pattern. It will become part of the context / instructions for the prompt.
 
-Consider the power to tailoring a prompt to your specific operating system:
+Consider the power of tailoring a prompt to your specific operating system:
 
-```
+```plaintext
 As a system administration on a $(uname -v) platform what is the best way to [DO_SOMETHING]
 ```
 
-or insert content from a file in your home directory:
+Or insert content from a file in your home directory:
 
-```
+```plaintext
 Given the following constraints $(cat ~/3_laws_of_robotics.txt) determine the best way to instruct my roomba to clean my kids room.
 ```
 
+#### Shell Command Safety
+
+The catchphrase "the prompt is the code" within AIA means that you have the power to execute any command you want, but you must be careful not to execute commands that could cause harm. AIA is not going to protect you from doing something stupid. Sure that's a copout. I just can't think (actually I can) of all the ways you can mess things up writing code. Remember what we learned from Forrest Gump "Stupid is as stupid does." So don't do anything stupid. If someone gives you a prompt as says "run this with AIA" you had better review the prompt before processing it.
+
 #### Chat Session Use
 
-When you use the `--shell` option to start a chat session, shell integration is available in your follow up prompts.  Suppose you started up a chat session using a roll of "Ruby Expert" expecting to chat about changes that could be made to a specific class BUT you forgot to include the class source file as part of the context when you got started.  You could enter this as your follow up prompt to this to keep going:
+When you use the `--shell` option to start a chat session, shell integration is available in your follow up prompts. Suppose you started a chat session (--chat) using a role of "Ruby Expert" expecting to chat about changes that could be made to a specific class BUT you forgot to include the class source file as part of the context when you got started. You could enter this as your follow up prompt to keep going:
 
-```
+```plaintext
 The class I want to chat about refactoring is this one: $(cat my_class.rb)
 ```
 
-That inserts the entire class source file into your follow up prompt.  You can continue chatting with you AI Assistant avout changes to the class.
+That inserts the entire class source file into your follow up prompt. You can continue chatting with you AI Assistant about changes to the class.
 
 ## *E*mbedded *R*u*B*y (ERB)
 
-The inclusion of dynamic content through the shell integration provided by the `--shell` option is significant.  `aia` also provides the full power of embedded Ruby code processing within the prompt text.
+The inclusion of dynamic content through the shell integration provided by the `--shell` option is significant. AIA also provides the full power of embedded Ruby code processing within the prompt text.
 
 The `--erb` option turns the prompt text file into a fully functioning ERB template. The [Embedded Ruby (ERB) template syntax (2024)](https://bophin-com.ngontinh24.com/article/language-embedded-ruby-erb-template-syntax) provides a good overview of the syntax and power of ERB.
 
-Most websites that have information about ERB will give examples of how to use ERB to generate dynamice HTML content for web-based applications.  That is a common use case for ERB.  `aia` on the other hand uses ERB to generate dynamic prompt text.
-
-### Chat Session Behavior
+Most websites that have information about ERB will give examples of how to use ERB to generate dynamic HTML content for web-based applications. That is a common use case for ERB. AIA on the other hand uses ERB to generate dynamic prompt text for LLM processing.
 
-In a chat session whether started by the `--chat` option or its equivalent with a directive within a prompt text file behaves a little differently w/r/t its binding and local variable assignments.  Since a chat session by definition has multiple prompts, setting a local variable in one prompt and expecting it to be available in a subsequent prompt does not work.  You need to use instance variables to accomplish this prompt to prompt carry over of information.
-
-Also since follow up prompts are expected to be a single thing - sentence or paragraph - terminated by a single return, its likely that ERB enhance will be of benefit; but, you may find a use for it.
 
 ## Prompt Directives
 
-Downstream processing directives were added to the `prompt_manager` gem used by `au` at version 0.4.1.  These directives are lines in the prompt text file that begin with "//" having this pattern:
+Downstream processing directives were added to the `prompt_manager` gem used by AIA at version 0.4.1. These directives are lines in the prompt text file that begin with "//" having this pattern:
 
-```
-//command parameters
+```bash
+//command params
 ```
 
-There is no space between the "//" and the command.
+There is no space between the "//" and the command. Commands do not have to have params. These params are typically space delimited when more than one is required. It all depens on the command.
 
-### Parameter and Shell Substitution in Directives 
+### Parameter and Shell Substitution in Directives
 
 When you combine prompt directives with prompt parameters and shell envar substitutions you can get some powerful compositional prompts.
 
 Here is an example of a pure generic directive.
 
-```
+```bash
 //[DIRECTIVE_NAME] [DIRECTIVE_PARAMS]
 ```
 
 When the prompt runs, you will be asked to provide a value for each of the parameters.  You could answer "shell" for the directive name and "calc 22/7" if you wanted a bad approximation of PI.
 
 Try this prompt file:
-```
+```bash
 //shell calc [FORMULA]
 
 What does that number mean to you?
 ```
 
-### `aia` Specific Directive Commands
+### Directive Syntax
+
+Directives can be entered in chat or prompt files using the following syntax:
+- `//command args`
 
-At this time `aia` only has a few directives which are detailed below.
+Supported directives:
+- `help`: Show available directives
+- `shell` or `sh`: Execute a shell command
+- `ruby` or `rb`: Execute Ruby code
+- `config` or `cfg`: Show or update configuration
+- `include` or `inc`: Include file content
+- `next`: Set/Show the next prompt ID to be processed
+- `pipeline`: Set/Extend/Show the workflow of prompt IDs
+
+When a directive produces output, it is added to the chat context. If there is no output, you are prompted again.
+
+### AIA Specific Directive Commands
+
+At this time AIA only has a few directives which are detailed below.
 
 #### //config
 
@@ -211,25 +342,23 @@ The switch options are treated like booleans.  They are either `true` or `false`
 
 To set the value of a switch using ``//config` for example `--terse` or `--chat` to this:
 
-```
+```bash
 //config chat? = true
 //config terse? = true
 ```
 
 A configuration item such as `--out_file` or `--model` has an associated value on the command line.  To set that value with the `//config` directive do it like this:
 
-```
+```bash
 //config model = gpt-3.5-turbo
 //config out_file = temp.md
-//config backend = mods
 ```
 
 BTW: the "=" is completely options.  Its actuall ignored as is ":=" if you were to choose that as your assignment operator.  Also the number of spaces between the item and the value is complete arbitrary.  I like to line things up so this syntax is just as valie:
 
-```
+```bash
 //config model       gpt-3.5-turbo
 //config out_file    temp.md
-//config backend     mods
 //config chat?       true
 //config terse?      true
 //config model       gpt-4
@@ -240,7 +369,7 @@ NOTE: if you specify the same config item name more than once within the prompt
 #### //include
 
 Example:
-```
+```bash
 //include path_to_file
 ```
 
@@ -248,100 +377,120 @@ The `path_to_file` can be either absolute or relative.  If it is relative, it is
 
 The file that is included will have any comments or directives excluded.  It is expected that the file will be a text file so that its content can be pre-pended to the existing prompt; however, if the file is a source code file (ex: file.rb) the source code will be included HOWEVER any comment line or line that starts with "//" will be excluded.
 
-TODO:  Consider adding a command line option `--include_dir` to specify the place from which relative files are to come.
 
 #### //ruby
-Example:
-```
-//ruby any_code_that_returns_an_instance_of_String
-```
 
-This directive is in addition to ERB.  At this point the `//ruby` directive is limited by the current binding which is within the `AIA::Directives#ruby` method.  As such it is not likely to see much use.
+The `//ruby` directive executes Ruby code. You can use this to perform complex operations or interact with Ruby libraries.
 
-However, sinces it implemented as a simple `eval(code)` then there is a potential for use like this:
-```
-//ruby load(some_ruby_file); execute_some_method
+For example:
+```ruby
+//ruby puts "Hello from Ruby"
 ```
 
-Each execution of a `//ruby` directive will be a fresh execution of the `AIA::Directives#ruby` method so you cannot carry local variables from one invocation to another; however, you could do something with instance variables or global variables.  You might even add something to the `AIA.config` object to be pasted on to the next invocation of the directive within the context of the same prompt.
+You can also use the `--rq` option to specify Ruby libraries to require before executing Ruby code:
+
+```bash
+# Command line
+aia --rq json,csv my_prompt
+
+# In chat
+//ruby JSON.parse('{"data": [1,2,3]}')["data"]
+```
 
 #### //shell
 Example:
-```
+```bash
 //shell some_shell_command
 ```
 
 It is expected that the shell command will return some text to STDOUT which will be pre-pending to the existing prompt text within the prompt file.
 
 There are no limitations on what the shell command can be.  For example if you wanted to bypass the stripping of comments and directives from a file you could do something like this:
-```
+```bash
 //shell cat path_to_file
 ```
 
 Which does basically the same thing as the `//include` directive, except it uses the entire content of the file.  For relative file paths the same thing applies.  The file's path will be relative to the PWD.
 
 
+#### //next
+Examples:
+```bash
+# Show the next promt ID
+//next
+
+# Set the next prompt ID
+//next prompt_id
 
-### Backend Directive Commands
+# Same as
+//config next
+//config next = prompt_id
+```
 
-See the source code for the directives supported by the backends which at this time are configuration-based as well.
+#### //pipeline
 
-- [mods](lib/aia/tools/mods.rb)
-- [sgpt](lib/aia/tools/sgpt.rb)
+Examples:
+```bash
+# Show the current prompt workflow
+//pipeline
 
-For example `mods` has a configuration item `topp` which can be set by a directive in a prompt text file directly.
+# Set the prompt workflow
+//pipeline =  prompt_id_1, prompt_id_2, prompt_id_3
 
-```
-//topp 1.5
-```
+# Extend the prompt workflow
+//pipeline <<  prompt_id_4, prompt_id_5, prompt_id_6
 
-If `mods` is not the backend the `//topp` direcive is ignored.
+# Same as
+//config pipeline
+//config pipeline = prompt_id_1, prompt_id_2, prompt_id_3
+//config pipeline <<  prompt_id_4, prompt_id_5, prompt_id_6
+```
 
 ### Using Directives in Chat Sessions
 
-Whe you are in a chat session, you may use a directive as a follow up prompt.  For example if you started the chat session with the option `--terse` expecting to get short answers from the backend; but, then you decide that you want more comprehensive answers you may do this:
+Whe you are in a chat session, you may use a directive as a follow up prompt.  For example if you started the chat session with the option `--terse` expecting to get short answers from the LLM; but, then you decide that you want more comprehensive answers you may do this in a chat follow up:
 
-```
+```bash
 //config terse? false
 ```
 
-The directive is executed and a new follow up prompt can be entered with a more lengthy response generated from the backend.
+The directive is executed and a new follow up prompt can be entered with a more lengthy response generated from the LLM.
 
+The directive `//clear` truncates your entire session context. The LLM will not remember anything you have discussed.
 
 ## Prompt Sequences
 
-Why would you need/want to use a sequence of prompts in a batch situation.  Maybe you have a complex prompt which exceeds the token limitations of your model for input so you need to break it up into multiple parts.  Or suppose its a simple prompt but the number of tokens on the output is limited and you do not get exactly the kind of full response for which you were looking.
+Why would you need/want to use a sequence of prompts in a batch situation. Maybe you have a complex prompt which exceeds the token limitations of your model for input so you need to break it up into multiple parts. Or suppose its a simple prompt but the number of tokens on the output is limited and you do not get exactly the kind of full response for which you were looking.
 
-Sometimes it takes a series of prompts to get the kind of response that you want.  The reponse from one prompt becomes a context for the next prompt.  This is easy to do within a `chat` session were you are manually entering and adjusting your prompts until you get the kind of response that you want.
+Sometimes it takes a series of prompts to get the kind of response that you want.  The reponse from one prompt becomes a context for the next prompt. This is easy to do within a `chat` session were you are manually entering and adjusting your prompts until you get the kind of response that you want.
 
-If you need to do this on a regular basis or within a batch you can use `aia` and the `--next` and `--pipeline` command line options.
+If you need to do this on a regular basis or within a batch you can use AIA and the `--next` and `--pipeline` command line options.
 
-These two options specify the sequence of prompt IDs to be processed. Both options are available to be used within a prompt file using the `//config` directive.  Like all embedded directives you can take advantage of parameterization shell integration and Ruby.  I'm start to feel like TIm Tool man - more power!
+These two options specify the sequence of prompt IDs to be processed. Both options are available to be used within a prompt file using the `//next` and `//pipeline` directives. Like all embedded directives you can take advantage of parameterization shell integration and Ruby.  With this kind of dynamic content and control flow in your prompts you will start to feel like Tim the Tool man - more power!
 
 Consider the condition in which you have 4 prompt IDs that need to be processed in sequence.  The IDs and associated prompt file names are:
 
-| Promt ID | Prompt File |
-| -------- | ----------- |
-| one.     | one.txt     |
-| two.     | two.txt     |
-| three.   | three.txt   |
-| four.    | four.txt    |
+| Prompt ID | Prompt File |
+| --------- | ----------- |
+| one       | one.txt     |
+| two       | two.txt     |
+| three     | three.txt   |
+| four      | four.txt    |
 
 
 ### --next
 
-```shell
-export AIA_OUT_FILE=temp.md 
-aia one --next two
-aia three --next four temp.md
+```bash
+aia one --next two --out_file temp.md
+aia three --next four temp.md -o answer.md
 ```
 
-or within each of the prompt files you use the config directive:
+or within each of the prompt files you use the `//next` directive:
 
-```
-one.txt contains //config next two
-two.txt contains //config next three
-three.txt contains //config next four
+```bash
+one.txt contains //next two
+two.txt contains //next three
+three.txt contains //next four
 ```
 BUT if you have more than two prompts in your sequence then consider using the --pipeline option.
 
@@ -349,11 +498,15 @@ BUT if you have more than two prompts in your sequence then consider using the -
 
 ### --pipeline
 
-`aia one --pipeline two,three,four`
+```bash
+aia one --pipeline two,three,four
+```
 
 or inside of the `one.txt` prompt file use this directive:
 
-`//config pipeline two,three,four`
+```bash
+//pipeline two,three,four
+```
 
 **The directive //pipeline is short for //config pipeline**
 
@@ -363,15 +516,15 @@ Since the response of one prompt is fed into the next prompt within the sequence
 
 
 | Prompt File | Directive |
-| --- | --- |
-| one.txt | //config out_file one.md |
-| two.txt | //config out_file two.md |
-| three.txt | //config out_file three.md |
-| four.txt | //config out_file four.md |
+| ----------- | --------- |
+| one.txt     | //config out_file one.md |
+| two.txt     | //config out_file two.md |
+| three.txt   | //config out_file three.md |
+| four.txt    | //config out_file four.md |
 
 This way you can see the response that was generated for each prompt in the sequence.
 
-### Example pipline
+### Example pipeline
 
 TODO: the audio-to-text is still under development.
 
@@ -379,24 +532,24 @@ Suppose you have an audio file of a meeting.  You what to get a transcription of
 
 Create two prompts named transcribe.txt and tech_summary.txt
 
-```
+```bash
 # transcribe.txt
 # Desc: takes one audio file
 # note that there is no "prompt" text only the directive
 
-//config backend  client
 //config model    whisper-1
 //next            tech_summary
 ```
+
 and
 
-```
+```bash
 # tech_summary.txt
 
-//config model    gpt-4-turbo
+//config model    gpt-4o-mini
 //config out_file meeting_summary.md
 
-Review the raw transcript of a technical meeting, 
+Review the raw transcript of a technical meeting,
 summarize the discussion and
 note any action items that were generated.
 
@@ -405,7 +558,7 @@ Format your response in markdown.
 
 Now you can do this:
 
-```
+```bash
 aia transcribe my_tech_meeting.m4a
 ```
 
@@ -414,104 +567,62 @@ You summary of the meeting is in the file `meeting_summary.md`
 
 ## All About ROLES
 
-### The --roles_dir (AIA_ROLES_DIR)
+### The --roles_prefix (AIA_ROLES_PREFIX)
 
-There are two kinds of prompts
-1. instructional - tells the LLM what to do
-2. personification - tells the LLM who it should pretend to be when it does its transformational work.
+The second kind of prompt is called a role (aka system prompt). Sometimes the role is incorporated into the instruction. For example, "As a magician make a rabbit appear out of a hat." To reuse the same role in multiple prompts, AIA encourages you to designate a special subdirectory for prompts that are specific to personification - roles.
 
-That second kind of prompt is called a role.  Sometimes the role is incorporated into the instruction.  For example "As a magician make a rabbit appear out of a hat."  To reuse the same role in multiple prompts `aia` encourages you to designate a special `roles_dir` into which you put prompts that are specific to personification - roles.
-
-The default `roles_dir` is a sub-directory of the `prompts_dir` named roles.  You can, however, put your `roles_dir` anywhere that makes sense to you.
+The default `roles_prefix` is set to 'roles'. This creates a subdirectory under the `prompts_dir` where role files are stored. Internally, AIA calculates a `roles_dir` value by joining `prompts_dir` and `roles_prefix`. It is recommended to keep the roles organized this way for better organization and management.
 
 ### The --role Option
 
-The `--role` option is used to identify a personification prompt within your roles directory which defines the context within which the LLM is to provide its response.  The text of the role ID is pre-pended to the text of the primary prompt to form a complete prompt to be processed by the backend.
+The `--role` option is used to identify a personification prompt within your roles directory which defines the context within which the LLM is to provide its response.  The text of the role ID is pre-pended to the text of the primary prompt to form a complete prompt to be processed by the LLM.
 
 For example consider:
 
-```shell
+```bash
 aia -r ruby refactor my_class.rb
 ```
 
-Within the roles directory the contents of the text file `ruby.txt` will be pre-pre-pended to the contents of the `refactor.txt` file from the prompts directory to produce a complete prompt.  That complete prompt will have any parameters followed by directives processed before sending the combined prompt text to the backend.
+The role ID is `ruby` the prompt ID is `refactor` and my_class.rb is a context file.
+
+Within the roles directory the contents of the text file `ruby.txt` will be pre-pre-pended to the contents of the `refactor.txt` file from the prompts directory to produce a complete prompt.  That complete prompt will have any parameters followed by directives processed before sending the combined prompt text and the content of the context file to the LLM.
 
 Note that `--role` is just a way of saying add this prompt text file to the front of this other prompt text file.  The contents of the "role" prompt could be anything.  It does not necessarily have be an actual role.
 
-`aia` fully supports a directory tree within the `prompts_dir` as a way of organization or classification of your different prompt text files.
+AIA fully supports a directory tree within the `prompts_dir` as a way of organization or classification of your different prompt text files.
 
-```shell
-aia -r sw_eng doc_the_methods my_class.rb
+```bash
+aia -r ruby sw_eng/doc_the_methods my_class.rb
 ```
 
-In this example the prompt text file `$AIA_ROLES_DIR/sw_eng.txt` is prepended to the prompt text file `$AIA_PROMPTS_DIR/doc_the_methods.txt`
-
+In this example the prompt text file `$AIA_ROLES_PREFIX/ruby.txt` is prepended to the prompt text file `$AIA_PROMPTS_DIR/sw_eng/doc_the_methods.txt`
 
 ### Other Ways to Insert Roles into Prompts
 
-Since `aia` supports parameterized prompts you could make a keyword like "[ROLE]" be part of your prompt.  For example consider this prompt:
+Since AIA supports parameterized prompts you could make a keyword like "[ROLE]" be part of your prompt.  For example consider this prompt:
 
 ```text
 As a [ROLE] tell me what you think about [SUBJECT]
 ```
 
-When this prompt is processed, `aia` will ask you for a value for the keyword "ROLE" and the keyword "SUBJECT" to complete the prompt.  Since `aia` maintains a history of your previous answers, you could just choose something that you used in the past or answer with a completely new value.
+When this prompt is processed, AIA will ask you for a value for the keyword "[ROLE]" and the keyword "[SUBJECT]" to complete the prompt.  Since AIA maintains a history of your previous answers, you could just choose something that you used in the past or answer with a completely new value.
 
 ## External CLI Tools Used
 
-To install the external CLI programs used by aia:
-  
-  brew install fzf mods rg glow
+To install the external CLI programs used by AIA:
+
+  brew install fzf
 
 fzf
   Command-line fuzzy finder written in Go
   [https://github.com/junegunn/fzf](https://github.com/junegunn/fzf)
 
-mods
-  AI on the command-line
-  [https://github.com/charmbracelet/mods](https://github.com/charmbracelet/mods)
-
-rg
-  Search tool like grep and The Silver Searcher
-  [https://github.com/BurntSushi/ripgrep](https://github.com/BurntSushi/ripgrep)
-
-glow
-  Render markdown on the CLI
-  [https://github.com/charmbracelet/glow](https://github.com/charmbracelet/glow)
-
-A text editor whose executable is setup in the
-system environment variable 'EDITOR' like this:
-
-  export EDITOR="subl -w"
-
-### Optional External CLI-tools
-
-#### Backend Processor `llm`
-
-```
-llm  Access large language models from the command-line
-     |   brew install llm
-     |__ https://llm.datasette.io/
-```
-
-As of `aia v0.5.13` the `llm` backend processor is available in a limited integration.  It is a very powerful python-based implementation that has its own prompt templating system.  The reason that it is be included within the `aia` environment is for its ability to make use of local LLM models.
-
-
-#### Backend Processor `sgpt`
-
-`shell-gpt` aka `sgpt` is also a python implementation of a CLI-tool that processes prompts through OpenAI.  It has less features than both `mods` and `llm` and is less flexible.
-
-#### Occassionally Useful Tool `plz`
-
-`plz-cli` aka `plz` is not integrated with `aia` however, it gets an honorable mention for its ability to except a prompt that tailored to doing something on the command line.  Its response is a CLI command (sometimes a piped sequence) that accomplishes the task set forth in the prompt.  It will return the commands to be executed agaist the data files you specified with a query to execute the command.
-
-- brew install plz-cli
 
 ## Shell Completion
 
 You can setup a completion function in your shell that will complete on the prompt_id saved in your `prompts_dir` - functions for `bash`, `fish` and `zsh` are available.  To get a copy of these functions do this:
 
-```shell
+```bash
 aia --completion bash
 ```
 
@@ -523,29 +634,30 @@ Copy the function to a place where it can be installed in your shell's instance.
 
 This is just between you and me so don't go blabbing this around to everyone.  My most power prompt is in a file named `ad_hoc.txt`. It looks like this:
 
-> [WHAT NOW HUMAN]
+```text
+[WHAT_NOW_HUMAN]
+```
 
 Yep.  Just a single parameter for which I can provide a value of anything that is on my mind at the time.  Its advantage is that I do not pollute my shell's command history with lots of text.
 
-Which do you think is better to have in your shell's history file?
-
-```shell
-mods "As a certified public accountant specializing in forensic audit and analysis of public company financial statements, what do you think of mine?  What is the best way to hide the millions dracma that I've skimmed?"  < financial_statement.txt
+```bash
+aia ad_hoc
 ```
 
-or
+Or consider this executable prompt file:
 
-```shell
-aia ad_hoc financial_statement.txt
+```bash
+#!/usr/bin/env aia run
+[WHAT_NOW_HUMAN]
 ```
 
-Both do the same thing; however, `aia` does not put the text of the prompt into the shell's history file.... of course the keyword/parameter value is saved in the prompt's JSON file and the prompt with the response are logged unless `--no-log` is specified; but, its not messing up the shell history!
+Where the `run` prompt ID has a `run.txt` file in the prompt directory that is basically empty.  Or maybe `run.txt` has some prompt instructions for how to run the prompt - some kind of meta-thinking instructions.
 
 ## My Configuration
 
 I use the `bash` shell.  In my `.bashrc` file I source another file named `.bashrc__aia` which looks like this:
 
-```shell
+```bash
 # ~/.bashic_aia
 # AI Assistant
 
@@ -553,43 +665,91 @@ I use the `bash` shell.  In my `.bashrc` file I source another file named `.bash
 export AIA_PROMPTS_DIR=~/.prompts
 export AIA_OUT_FILE=./temp.md
 export AIA_LOG_FILE=$AIA_PROMPTS_DIR/_prompts.log
-export AIA_BACKEND=mods
-export AIA_MODEL=gpt-4-1106-preview
+export AIA_MODEL=gpt-4o-mini
 
-# Not a default.  Invokes spinner.
+# Not a default.  Invokes spinner.  If not true then there is no spinner
+# for feedback while waiting for the LLM to respond.
 export AIA_VERBOSE=true
 
-alias chat='aia chat --terse'
+alias chat='aia --chat --shell --erb --terse'
 
 # rest of the file is the completion function
 ```
 
-Here is what my `chat` prompt file looks like:
 
-```shell
-# ~/.prompts/chat.txt
-# Desc: Start a chat session
 
-//config chat? = true
+## Executable Prompts
+
+With all of the capabilities of the AI Assistant, you can create your own executable prompts. These prompts can be used to automate tasks, generate content, or perform any other action that you can think of.  All you need to get started with executable prompts is a prompt that does not do anything.  For example consider my `run.txt` prompt.
 
-[WHAT]
+```bash
+# ~/.prompts/run.txt
+# Desc: Run executable prompts coming in via STDIN
 ```
 
+Remember that the '#' character indicates a comment line making the `run` prompt ID basically a do nothing prompt.
+
+An executable prompt can reside anywhere either in your $PATH or not.  That is your choice.  It must however be executable.  Consider the following `top10` executable prompt:
+
+```bash
+#!/usr/bin/env aia run --no-out_file
+# File: top10
+# Desc: The tope 10 cities by population
+
+what are the top 10 cities by population in the USA. Summarize what people
+like about living in each city. Include an average cost of living. Include
+links to the Wikipedia pages.  Format your response as a markdown document.
+```
+
+Make sure that it is executable.
+
+```bash
+chmod +x top10
+```
+
+The magic is in the first line of the prompt.  It is a shebang line that tells the system how to execute the prompt.  In this case it is telling the system to use the `aia` command line tool to execute the `run` prompt.  The `--no-out_file` option tells the AIA command line tool not to write the output of the prompt to a file.  Instead it will write the output to STDOUT.  The remaining content of this `top10` prompt is send via STDIN to the LLM.
+
+Now just execute it like any other command in your terminal.
+
+```bash
+./top10
+```
+
+Since its output is going to STDOUT you can setup a pipe chain.  Using the CLI program `glow` to render markdown in the terminal
+(brew install glow)
+
+```bash
+./top10 | glow
+```
+
+This executable prompt concept sets up the building blocks of a *nix CLI-based pipeline in the same way that the --pipeline and --next options and directives are used.
+
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+**ShellCommandExecutor Refactor:**
+The `ShellCommandExecutor` is now a class (previously a module). It stores the config object as an instance variable and provides cleaner encapsulation. For backward compatibility, class-level methods are available and delegate to instance methods internally.
+
+**Prompt Variable Fallback:**
+When processing a prompt file without a `.json` history file, variables are always parsed from the prompt text so you are prompted for values as needed.
 
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/MadBomber/aia.
 
-I've designed `aia` so that it should be easy to integrate other backend LLM processors.  If you've found one that you like, send me a pull request or a feature request.
+When you find problems with AIA please note them as an issue.  This thing was written mostly by a human and you know how error prone humans are.  There should be plenty of errors to find.
+
+I'm not happy with the way where some command line options for external command are hard coded.  I'm specific talking about the way in which the `rg` and `fzf` tools are used.  Their options decide the basic look and feel of the search capability on the command line.  Maybe they should be part of the overall configuration so that users can tune their UI to the way they like.
+
+## History of Development
+
+I originally wrote a tiny script called `aip.rb` to experiment with parameterized prompts. That was in August of 2023. AIP meant AI Parameterized. Adding an extra P for Prompts just seemed to be a little silly. It lived in my [scripts repo](https://github.com/MadBomber/scripts) for a while. It became useful to me so of course I need to keep enhancing it. I moved it into my [experiments repo](https://github.com/MadBomber/experiments) and began adding features in a haphazard manner. No real plan or architecture. From those experiments I refactored out the [prompt_manager gem](https://github.com/MadBomber/prompt_manager) and the [ai_client gem](https://github.com/MadBomber/ai_client)(https://github.com/MadBomber/ai_client). The name was changed from AIP to AIA and it became a gem.
 
-When you find problems with `aia` please note them as an issue.  This thing was written mostly by a human and you know how error prone humans are.  There should be plenty of errors to find.
+All of that undirected experimentation without a clear picture of where this thing was going resulted in chaotic code. I would use an Italian food dish to explain the organization but I think chaotic is more descriptive.
 
-I've been thinking that the REGEX used to identify a keyword within a prompt could be a configuration item.  I chose to use square brackets and uppercase in the default regex; maybe, you have a collection of prompt files that use some other regex.  Why should it be one way and not the other.
+## Roadmap
 
-Also I'm not happy with the way where I a some command line options for external command hard coded.  I think they should be part of the configuration as well.  For example the way I'm using `rg` and `fzf` may not be the way that you want to use them.
+- support for using Ruby-based functional callback tools
+- support for Model Context Protocol
 
 ## License