prompt_manager 0.5.7 → 0.5.8

data/README.md

@@ -1,58 +1,92 @@
 # PromptManager
 
-Manage the parameterized prompts (text) used in generative AI (aka chatGPT, OpenAI, _et.al._) using storage adapters such as FileSystemAdapter and ActiveRecordAdapter.
-
-**Current Version**: 0.5.3
-
-**Breaking Change** in version 0.3.0 - The value of the parameters Hash for a keyword is now an Array instead of a single value.  The last value in the Array is always the most recent value used for the given keyword.  This was done to support the use of a Readline::History object editing in the [aia](https://github.com/MadBomber/aia) CLI tool
-### Latest Capabilities
-<!-- Tocer[start]: Auto-generated, don't remove. -->
+> [!CAUTION]
+> ## ⚠️ Breaking Changes are Coming ⚠️
+> See [Roadmap](#roadmap) for details about upcoming changes.
+<br />
+<div align="center">
+  <table>
+    <tr>
+      <td width="40%" align="center" valign="top">
+        <a href="https://madbomber.github.io/blog/" target="_blank">
+          <img src="prompt_manager_logo.png" alt="PromptManager - The Enchanted Librarian of AI Prompts" width="800">
+        </a>
+        <br /><br />
+        [Comprehensive Documentation Website](https://madbomber.github.io/prompt_manager/)
+      </td>
+      <td width="60%" align="left" valign="top">
+        Like an enchanted librarian organizing floating books of knowledge, PromptManager helps you masterfully orchestrate and organize your AI prompts through wisdom and experience. Each prompt becomes a living entity that can be categorized, parameterized, and interconnected with golden threads of relationships.
+        <br/><br/>
+        <h3>Key Features</h3>
+        <ul>
+            <li><strong>📚 <a href="#storage-adapters">Multiple Storage Adapters</a></strong>
+            <li><strong>🔧 <a href="#parameterized-prompts">Parameterized Prompts</a></strong>
+            <li><strong>📋 <a href="#directive-processing">Directive Processing</a></strong>
+            <li><strong>🎨 <a href="#erb-and-shell-integration">ERB Integration</a></strong>
+            <li><strong>🌍 <a href="#erb-and-shell-integration">Shell Integration</a></strong>
+            <li><strong>📖 <a href="#comments-and-documentation">Inline Documentation</a></strong>
+            <li><strong>📊 <a href="#parameter-history">Parameter History</a></strong>
+            <li><strong>⚡ <a href="#error-handling">Error Handling</a></strong>
+            <li><strong>🔌 <a href="#extensible-architecture">Extensible Architecture</a></strong>
+        </ul>
+      </td>
+    </tr>
+  </table>
+</div>
 
 ## Table of Contents
 
-    - [Latest Capabilities](#latest-capabilities)
-  - [Installation](#installation)
-  - [Usage](#usage)
-  - [Overview](#overview)
-    - [Prompt Initialization Options](#prompt-initialization-options)
-    - [Generative AI (gen-AI)](#generative-ai-gen-ai)
-      - [What does a keyword look like?](#what-does-a-keyword-look-like)
-      - [All about directives](#all-about-directives)
-        - [Example Prompt with Directives](#example-prompt-with-directives)
-        - [Accessing and Setting Parameter Values](#accessing-and-setting-parameter-values)
-        - [Dynamic Directives](#dynamic-directives)
-        - [Executing Directives](#executing-directives)
-      - [Comments Are Ignored](#comments-are-ignored)
-  - [Storage Adapters](#storage-adapters)
-    - [FileSystemAdapter](#filesystemadapter)
-      - [Configuration](#configuration)
-        - [prompts_dir](#prompts_dir)
-        - [search_proc](#search_proc)
-        - [File Extensions](#file-extensions)
-      - [Example Prompt Text File](#example-prompt-text-file)
-      - [Example Prompt Parameters JSON File](#example-prompt-parameters-json-file)
-      - [Extra Functionality](#extra-functionality)
-    - [ActiveRecordAdapter](#activerecordadapter)
-      - [Configuration](#configuration-1)
-        - [model](#model)
-        - [id_column](#id_column)
-        - [text_column](#text_column)
-        - [parameters_column](#parameters_column)
-    - [Other Potential Storage Adapters](#other-potential-storage-adapters)
-  - [Development](#development)
-  - [Contributing](#contributing)
-  - [License](#license)
-
-<!-- Tocer[finish]: Auto-generated, don't remove. -->
-
-
-- **Directive Processing:** Processes directives such as `//include` (aliased as `//import`) with loop protection.
-- **ERB Processing:** Supports ERB templating within prompts.
-- **Environment Variable Embedding:** Automatically substitutes system environment variables in prompts.
-- **Improved Parameter Handling:** Refactored to maintain a history of parameter values and added error handling for parameter substitution.
-- **ActiveRecord Adapter:** Facilitates storing and retrieving prompts via an ActiveRecord model.
-- **Enhanced Error Handling:** Added specific error classes for better error handling, including StorageError, ParameterError, and ConfigurationError.
-- **Customizable Keyword Pattern:** Ability to customize the pattern used for detecting keywords in prompts.
+* [Installation](#installation)
+* [Quick Start](#quick-start)
+* [Core Features](#core-features)
+  * [Parameterized Prompts](#parameterized-prompts)
+    * [Keyword Syntax](#keyword-syntax)
+    * [Custom Patterns](#custom-patterns)
+    * [Working with Parameters](#working-with-parameters)
+  * [Directive Processing](#directive-processing)
+    * [Built\-in Directives](#built-in-directives)
+    * [Directive Syntax](#directive-syntax)
+    * [Custom Directive Processors](#custom-directive-processors)
+  * [ERB and Shell Integration](#erb-and-shell-integration)
+    * [ERB Templates](#erb-templates)
+    * [Environment Variables](#environment-variables)
+  * [Comments and Documentation](#comments-and-documentation)
+    * [Line Comments](#line-comments)
+    * [Block Comments](#block-comments)
+    * [Blank Lines](#blank-lines)
+  * [Parameter History](#parameter-history)
+  * [Error Handling](#error-handling)
+* [Storage Adapters](#storage-adapters)
+  * [FileSystemAdapter](#filesystemadapter)
+    * [Configuration](#configuration)
+    * [File Structure](#file-structure)
+    * [Custom Search](#custom-search)
+    * [Extra Methods](#extra-methods)
+  * [ActiveRecordAdapter](#activerecordadapter)
+    * [Configuration](#configuration-1)
+    * [Database Setup](#database-setup)
+  * [Custom Adapters](#custom-adapters)
+* [Configuration](#configuration-2)
+  * [Initialization Options](#initialization-options)
+  * [Global Configuration](#global-configuration)
+* [Advanced Usage](#advanced-usage)
+  * [Custom Keyword Patterns](#custom-keyword-patterns)
+  * [Dynamic Directives](#dynamic-directives)
+  * [Search Capabilities](#search-capabilities)
+* [Examples](#examples)
+  * [Basic Usage](#basic-usage)
+  * [With Search](#with-search)
+  * [Custom Storage](#custom-storage)
+* [Extensible Architecture](#extensible-architecture)
+  * [Extension Points](#extension-points)
+  * [Potential Extensions](#potential-extensions)
+* [Roadmap](#roadmap)
+  * [v0\.9\.0 \- Modern Prompt Format (Breaking Changes)](#v090---modern-prompt-format-breaking-changes)
+  * [v1\.0\.0 \- Stability Release](#v100---stability-release)
+  * [Future Enhancements](#future-enhancements)
+* [Development](#development)
+* [Contributing](#contributing)
+* [License](#license)
 
 ## Installation
 
@@ -64,271 +98,519 @@ If bundler is not being used to manage dependencies, install the gem by executin
 
     gem install prompt_manager
 
-## Usage
+## Quick Start
 
-See [examples/simple.rb](examples/simple.rb)
+```ruby
+require 'prompt_manager'
 
-See also [examples/using_search_proc.rb](examples/using_search_proc.rb)
+# Configure storage adapter
+PromptManager::Prompt.storage_adapter =
+  PromptManager::Storage::FileSystemAdapter.config do |config|
+    config.prompts_dir = '~/.prompts'
+  end.new
 
-## Overview
+# Load and use a prompt
+prompt = PromptManager::Prompt.new(id: 'greeting')
+prompt.parameters = {
+  "[NAME]" => "Alice",
+  "[LANGUAGE]" => "English"
+}
 
-### Prompt Initialization Options
-- `id`: A String name for the prompt.
-- `context`: An Array for additional context.
-- `directives_processor`: An instance of PromptManager::DirectiveProcessor (default), can be customized.
-- `external_binding`: A Ruby binding to be used for ERB processing.
-- `erb_flag`: Boolean flag to enable ERB processing in the prompt text.
-- `envar_flag`: Boolean flag to enable environment variable substitution in the prompt text.
+# Get the processed prompt text
+result = prompt.to_s
+```
 
-The `prompt_manager` gem provides functionality to manage prompts that have keywords and directives for use with generative AI processes.
+## Core Features
 
-### Generative AI (gen-AI)
+### Parameterized Prompts
 
-Gen-AI deals with the conversion (some would say execution) of a human natural language text (the "prompt") into something else using what are known as large language models (LLM) such as those available from OpenAI.  A parameterized prompt is one in which there are embedded keywords (parameters) which are place holders for other text to be inserted into the prompt.
+The heart of PromptManager is its ability to manage parameterized prompts - text templates with embedded keywords that can be replaced with dynamic values.
 
-The prompt_manager uses a regular expression to identify these keywords within the prompt. It uses the keywords as keys in a `parameters` Hash which is stored with the prompt text in a serialized form - for example as JSON.
+#### Keyword Syntax
 
-#### What does a keyword look like?
+By default, keywords are enclosed in square brackets: `[KEYWORD]`, `[MULTIPLE WORDS]`, or `[WITH_UNDERSCORES]`.
 
-By default, any text matching `[UPPERCASE_TEXT]` enclosed in square brackets is treated as a keyword. [KEYWORDS CAN ALSO HAVE SPACES] as well as the underscore character.
+```ruby
+prompt_text = "Hello [NAME], please translate this to [LANGUAGE]"
+```
 
-You can customize the keyword pattern by setting a different regular expression:
+#### Custom Patterns
+
+You can customize the keyword pattern to match your preferences:
 
 ```ruby
-# Use {{param}} style instead of [PARAM]
+# Use {{mustache}} style
 PromptManager::Prompt.parameter_regex = /(\{\{[A-Za-z_]+\}\})/
+
+# Use :colon style
+PromptManager::Prompt.parameter_regex = /(:[a-z_]+)/
 ```
 
-The regex must include capturing parentheses () to extract the keyword. The default regex is `/(\[[A-Z _|]+\])/`.
-#### All about directives
+The regex must include capturing parentheses `()` to extract the keyword.
 
-A directive is a line in the prompt text that starts with the two characters '//' - slash slash - just like in the old days of IBM JCL - Job Control Language.  A prompt can have zero or more directives.  Directives can have parameters and can make use of keywords.
+#### Working with Parameters
 
-The `prompt_manager` collects directives and provides a DirectiveProcessor class that currently implements the `//include` directive (also aliased as `//import`), which allows including content from other files with loop protection. It extracts keywords from directive lines and provides the substitution of those keywords with other text just like it does for the prompt.
+```ruby
+prompt = PromptManager::Prompt.new(id: 'example')
 
-##### Example Prompt with Directives
+# Get all keywords found in the prompt
+keywords = prompt.keywords  #=> ["[NAME]", "[LANGUAGE]"]
 
-Here is an example prompt text file with comments, directives and keywords:
+# Set parameter values
+prompt.parameters = {
+  "[NAME]" => "Alice",
+  "[LANGUAGE]" => "French"
+}
+
+# Get processed text with substitutions
+final_text = prompt.to_s
+
+# Save changes
+prompt.save
+```
+
+### Directive Processing
+
+Directives are special line oriented instructions in your prompts that begin with `//` starting in column 1. They're inspired by IBM JCL and provide powerful prompt composition capabilities.  A character string that begins with `//` but is not at the very beginning of the line will NOT be processed as a directive.
+
+#### Built-in Directives
+
+**`//include` (alias: `//import`)** - Include content from other files:
 
 ```text
-# prompts/sing_a_song.txt
-# Desc: Has the computer sing a song
+//include common/header.txt
+//import [TEMPLATE_NAME].txt
 
-//TextToSpeech [LANGUAGE] [VOICE NAME]
+Main prompt content here...
+```
 
-Say the lyrics to the song [SONG NAME].  Please provide only the lyrics without commentary.
+Features:
+- Loop protection prevents circular includes
+- Supports keyword substitution in file paths
+- Processes included files recursively
 
-__END__
-Computers will never replace Frank Sinatra
+#### Directive Syntax
+
+```text
+//directive_name [PARAM1] [PARAM2] options
+
+# Dynamic directives using keywords
+//[COMMAND] [OPTIONS]
+```
+
+#### Custom Directive Processors
+
+You can create custom directive processors:
+
+```ruby
+class MyDirectiveProcessor < PromptManager::DirectiveProcessor
+  def process_directive(directive, prompt)
+    case directive
+    when /^\/\/model (.+)$/
+      set_model($1)
+    when /^\/\/temperature (.+)$/
+      set_temperature($1.to_f)
+    else
+      super
+    end
+  end
+end
+
+prompt = PromptManager::Prompt.new(
+  id: 'example',
+  directives_processor: MyDirectiveProcessor.new
+)
 ```
 
-##### Accessing and Setting Parameter Values
+### ERB and Shell Integration
 
-Getting and setting keywords from a prompt is straightforward:
+#### ERB Templates
+
+Enable ERB processing for dynamic content generation:
 
 ```ruby
-prompt = PromptManager::Prompt.new(id: 'some_id')
-prompt.keywords   #=> an Array of keywords found in the prompt text
+prompt = PromptManager::Prompt.new(
+  id: 'dynamic',
+  erb_flag: true
+)
+```
 
-# Update the parameters hash with keyword values
-prompt.parameters = {
-  "[KEYWORD1]" => "value1",
-  "[KEYWORD2]" => "value2"
-}
+Example prompt with ERB:
 
-# Build the final prompt text
-# This substitutes values for keywords and processes directives
-# Comments are removed and the result is ready for the LLM
-final_text = prompt.to_s
+```text
+Today's date is <%= Date.today %>
+<% 5.times do |i| %>
+  Item <%= i + 1 %>
+<% end %>
+```
 
-# Save any parameter changes back to storage
-prompt.save
+#### Environment Variables
+
+Enable automatic environment variable substitution:
+
+```ruby
+prompt = PromptManager::Prompt.new(
+  id: 'with_env',
+  envar_flag: true
+)
 ```
 
-Internally, directives are stored in a Hash where each key is the full directive line (including the // characters) and the value is the result string from executing that directive. The directives are processed in the order they appear in the prompt text.
+Environment variables are automatically replaced in the prompt text.
+
+### Comments and Documentation
+
+PromptManager supports comprehensive inline documentation:
 
-##### Dynamic Directives
+#### Line Comments
 
-Since directies are collected after the keywords in the prompt have been substituted for their values, it is possible to have dynamically generated directives as part of a prompt.  For example:
+Lines beginning with `#` are treated as comments:
 
 ```text
-//[COMMAND] [OPTIONS]
-# or
-[SOMETHING]
+# This is a comment
+# Description: This prompt does something useful
+
+Actual prompt text here...
 ```
-... where [COMMAND] gets replaced by some directive name.  [SOMETHING] could be replaced by "//directive options"
 
-##### Executing Directives
+#### Block Comments
 
-The `prompt_manager` gem provides a basic DirectiveProcessor class that handles the `//include` directive (aliased as `//import`), which adds the contents of a file to the prompt with loop protection to prevent circular includes.
+Everything after `__END__` is ignored:
 
-Additionally, you can extend with your own directives or downstream processes. Here are some ideas on how directives could be used in prompt downstream process:
+```text
+Main prompt content...
 
-- "//model gpt-5" could be used to set the LLM model to be used for a specific prompt.
-- "//backend mods" could be used to set the backend prompt processor on the command line to be the `mods` utility.
-- "//chat" could be used to send the prompts and then start up a chat session about the prompt and its response.
+__END__
+Development notes:
+- This section is completely ignored
+- Great for documentation
+- TODO items
+```
 
-Its all up to how your application wants to support directives or not.
+#### Blank Lines
 
+Blank lines are automatically removed from the final output.
 
-#### Comments Are Ignored
+### Parameter History
 
-The `prompt_manager` gem ignores comments.  A line that begins with the '#' - pound (aka hash) character - is a line comment.  Any lines that follow a line that is '__END__ at the end of a file are considered comments.  Basically the '__END__' the end of the file.  Nothing is process following that line.
+PromptManager maintains a history of parameter values (since v0.3.0):
 
-The gem also ignores blank lines.
+```ruby
+# Parameters are stored as arrays
+prompt.parameters = {
+  "[NAME]" => ["Alice", "Bob", "Charlie"]  # Charlie is most recent
+}
 
-## Storage Adapters
+# The last value is always the most recent
+current_name = prompt.parameters["[NAME]"].last
 
-A storage adapter is a class instance that ties the `PromptManager::Prompt` class to a storage facility that holds the actual prompts. Currently there are 2 storage adapters implemented: FileSystemAdapter and ActiveRecordAdapter.
+# Useful for:
+# - Implementing value history in UIs
+# - Providing dropdown selections
+# - Tracking parameter usage over time
+```
 
-The `PromptManager::Prompt` to support a small set of methods.  A storage adapter can provide "extra" class or instance methods that can be used through the Prompt class.  See the `test/prompt_manager/prompt_test.rb` for guidance on creating a new storage adapter.
+### Error Handling
+
+PromptManager provides specific error classes for better debugging:
+
+```ruby
+begin
+  prompt = PromptManager::Prompt.new(id: 'missing')
+rescue PromptManager::StorageError => e
+  # Handle storage-related errors
+  puts "Storage error: #{e.message}"
+rescue PromptManager::ParameterError => e
+  # Handle parameter substitution errors
+  puts "Parameter error: #{e.message}"
+rescue PromptManager::ConfigurationError => e
+  # Handle configuration errors
+  puts "Configuration error: #{e.message}"
+end
+```
+
+## Storage Adapters
+
+Storage adapters provide the persistence layer for prompts. PromptManager includes two built-in adapters and supports custom implementations.
 
 ### FileSystemAdapter
 
-This is the first storage adapter developed. It saves prompts as text files within the file system inside a designated `prompts_dir` (directory) such as `~/.prompts` or where it makes the most sense to you.  Another example would be to have your directory on a shared file system so that others can use the same prompts.
+Stores prompts as text files in a directory structure.
 
-The `prompt ID` is the basename of the text file. For example `todo.txt` is the file for the prompt ID `todo` (see the examples directory.)
+#### Configuration
 
-The parameters for the `todo` prompt ID are saved in the same directory as `todo.txt` in a JSON file named `todo.json` (also in the examples directory.)
+```ruby
+PromptManager::Storage::FileSystemAdapter.config do |config|
+  config.prompts_dir       = "~/.prompts"      # Required
+  config.search_proc       = nil               # Optional custom search
+  config.prompt_extension  = '.txt'            # Default
+  config.params_extension  = '.json'           # Default
+end
+```
+
+#### File Structure
+
+```
+~/.prompts/
+├── greeting.txt        # Prompt text
+├── greeting.json       # Parameters
+├── email/
+│   ├── welcome.txt
+│   └── welcome.json
+```
+
+#### Custom Search
+
+Integrate with external search tools:
+
+```ruby
+config.search_proc = ->(query) {
+  # Use ripgrep for fast searching
+  `rg -l "#{query}" #{config.prompts_dir}`.split("\n")
+}
+```
+
+#### Extra Methods
+
+- `list` - Returns array of all prompt IDs
+- `path(id)` - Returns Pathname to prompt file
+
+### ActiveRecordAdapter
+
+Stores prompts in a database using ActiveRecord.
 
 #### Configuration
 
-Use a `config` block to establish the configuration for the class.
+```ruby
+PromptManager::Storage::ActiveRecordAdapter.config do |config|
+  config.model              = PromptModel      # Your AR model
+  config.id_column          = :prompt_id       # Column for ID
+  config.text_column        = :content         # Column for text
+  config.parameters_column  = :params          # Column for parameters
+end
+```
+
+#### Database Setup
+
+```ruby
+class CreatePrompts < ActiveRecord::Migration[7.0]
+  def change
+    create_table :prompts do |t|
+      t.string :prompt_id, null: false, index: { unique: true }
+      t.text :content
+      t.json :params
+      t.timestamps
+    end
+  end
+end
+```
+
+### Custom Adapters
+
+Create your own storage adapter:
 
 ```ruby
-PromptManager::Storage::FileSystemAdapter.config do |o|
-  o.prompts_dir       = "path/to/prompts_directory"
-  o.search_proc       = nil     # default
-  o.prompt_extension  = '.txt'  # default
-  o.params_extension  = '.json' # default
+class RedisAdapter
+  def initialize(redis_client)
+    @redis = redis_client
+  end
+
+  def get(id)
+    prompt_text = @redis.get("prompt:#{id}:text")
+    parameters = JSON.parse(@redis.get("prompt:#{id}:params") || '{}')
+    [prompt_text, parameters]
+  end
+
+  def save(id, text, parameters)
+    @redis.set("prompt:#{id}:text", text)
+    @redis.set("prompt:#{id}:params", parameters.to_json)
+  end
+
+  def delete(id)
+    @redis.del("prompt:#{id}:text", "prompt:#{id}:params")
+  end
+
+  def list
+    @redis.keys("prompt:*:text").map { |k| k.split(':')[1] }
+  end
 end
 ```
 
-The `config` block returns `self` so that means you can do this to setup the storage adapter with the Prompt class:
+## Configuration
+
+### Initialization Options
+
+When creating a prompt instance:
 
 ```ruby
-PromptManager::Prompt
-  .storage_adapter =
-    PromptManager::Storage::FileSystemAdapter
-      .config do |config|
-        config.prompts_dir = 'path/to/prompts_dir'
-      end.new
+prompt = PromptManager::Prompt.new(
+  id: 'example',
+  context: ['additional', 'context'],
+  directives_processor: CustomProcessor.new,
+  external_binding: binding,
+  erb_flag: true,
+  envar_flag: true
+)
 ```
 
-##### prompts_dir
+Options:
+- `id` - Unique identifier for the prompt
+- `context` - Additional context array
+- `directives_processor` - Custom directive processor
+- `external_binding` - Ruby binding for ERB
+- `erb_flag` - Enable ERB processing
+- `envar_flag` - Enable environment variable substitution
 
-This is either a `String` or a `Pathname` object.  All file paths are maintained in the class as `Pathname` objects.  If you provide a `String` it will be converted.  Relative paths will be converted to absolute paths.
+### Global Configuration
 
-An `ArgumentError` will be raised when `prompts_dir` does not exist or if it is not a directory.
+Set the storage adapter globally:
 
-##### search_proc
+```ruby
+PromptManager::Prompt.storage_adapter = adapter_instance
+```
 
-The default for `search_proc` is nil which means that the search will be preformed by a default `search` method which is basically reading all the prompt files to see which ones contain the search term. It will return an Array of prompt IDs for each prompt file found that contains the search term.  Its up to the application to select which returned prompt ID to use.
+## Advanced Usage
 
-There are faster ways to search and select files.  For example there are specialized search and selection utilities that are available for the command line. The `examples` directory contains a `bash` script named `rgfzf` that uses `rg` (aka `ripgrep`) to do the searching and `fzf` to do the selecting.
+### Custom Keyword Patterns
 
-See [examples/using_search_proc.rb](examples/using_search_proc.rb)
+Examples of different keyword patterns:
 
-##### File Extensions
+```ruby
+# Handlebars style: {{name}}
+PromptManager::Prompt.parameter_regex = /(\{\{[a-z_]+\}\})/
 
-These two configuration options are `String` objects that must start with a period "." utherwise an `ArgumentError` will be raised.
+# Colon prefix: :name
+PromptManager::Prompt.parameter_regex = /(:[a-z_]+)/
 
-* prompt_extension - default: '.txt'
-* params_extension - default: '.json'
+# Dollar sign: $NAME
+PromptManager::Prompt.parameter_regex = /(\$[A-Z_]+)/
 
-Currently the `FileSystemAdapter` only supports a JSON serializer for its parameters Hash.  Using any other values for these extensions will cause problems.
+# Percentage: %name%
+PromptManager::Prompt.parameter_regex = /(%[a-z_]+%)/
+```
 
-They exist so that there is a platform on to which other storage adapters can be built or serializers added.  This is not currently on the roadmap.
+### Dynamic Directives
 
-#### Example Prompt Text File
+Create directives that change based on parameters:
 
 ```text
-# ~/.prompts/joke.txt
-# Desc: Tell some jokes
+# Set directive name via parameter
+//[DIRECTIVE_TYPE] [OPTIONS]
 
-Tell me a few [KIND] jokes about [SUBJECT]
+# Conditional directives
+//include templates/[TEMPLATE_TYPE].txt
 ```
 
-Note the command lines at the top.  This is a convention I use.  It is not part of the software.  I find it helpful in documenting the prompt.
-
-#### Example Prompt Parameters JSON File
-
-```json
-{
-  "[KIND]": [
-    "pun",
-    "family friendly"
-  ],
-  "[SUBJECT]": [
-    "parrot",
-    "garbage man",
-    "snowman",
-    "weather girl"
-  ]
+### Search Capabilities
+
+Implement powerful search across prompts:
+
+```ruby
+# With FileSystemAdapter
+adapter.search_proc = ->(query) {
+  # Custom search implementation
+  results = []
+  Dir.glob("#{prompts_dir}/**/*.txt").each do |file|
+    content = File.read(file)
+    if content.include?(query)
+      results << File.basename(file, '.txt')
+    end
+  end
+  results
 }
+
+# With ActiveRecordAdapter
+PromptModel.where("content LIKE ?", "%#{query}%").pluck(:prompt_id)
 ```
 
-The last value in the keyword's Array is the most recent value used for that keyword.  This is a functionality established since v0.3.0.  Its purpose is to provide a history of values from which a user can select to repeat a previous value or to select ta previous value and edit it into something new.
+## Examples
 
-#### Extra Functionality
+### Basic Usage
 
-The `FileSystemAdapter` adds two new methods for use by the `Prompt` class:
+```ruby
+# examples/simple.rb
+require 'prompt_manager'
 
-- list - returns an Array of prompt IDs
-- path and path(prompt_id) - returns a `Pathname` object to the prompt file
+# Setup
+PromptManager::Prompt.storage_adapter =
+  PromptManager::Storage::FileSystemAdapter.config do |c|
+    c.prompts_dir = '~/.prompts'
+  end.new
 
-Use the `path(prompt_id)` form against the `Prompt` class
-Use `prompt.path` when you have an instance of a `Prompt`
+# Create and use a prompt
+prompt = PromptManager::Prompt.new(id: 'story')
+prompt.parameters = {
+  "[GENRE]" => "fantasy",
+  "[CHARACTER]" => "wizard"
+}
 
-### ActiveRecordAdapter
+puts prompt.to_s
+```
 
-The `ActiveRecordAdapter` assumes that there is a database already configured by the application program that is requiring `prompt_manager` which has a model that contains prompt content.  This model must have at least three columns which contain content for:
+### Advanced Integration with LLM and Streaming
 
-- a prompt ID
-- prompt text
-- prompt parameters
+See [examples/advanced_integrations.rb](examples/advanced_integrations.rb) for a complete example that demonstrates:
 
-The model and the columns for these three elements can have any name.  Those names are provided to the `ActiveRecordAdapter` in its config block.
+- **ERB templating** for dynamic content generation
+- **Shell integration** for environment variable substitution
+- **OpenAI API integration** with streaming responses
+- **Professional UI** with spinner feedback using `tty-spinner`
 
+This example shows how to create sophisticated AI prompts that adapt to your system environment and stream responses in real-time.
 
-#### Configuration
+### With Search
 
-Use a `config` block to establish the configuration for the class.
+See [examples/using_search_proc.rb](examples/using_search_proc.rb) for advanced search integration.
 
-The `PromptManager::Prompt` class expects an instance of a storage adapter class.  By convention storage adapter class config methods will return `self` so that a simple `new` after the config will establish the instance.
+### Custom Storage
 
 ```ruby
-PromptManager::Prompt
-  .storage_adapter =
-    PromptManager::Storage::ActiveRecordAdapter.config do |config|
-      config.model              = DbPromptModel # any ActiveRecord::Base model
-      config.id_column          = :prompt_name
-      config.text_column        = :prompt_text
-      config.parameters_column  = :prompt_params
-    end.new # adapters an instances of the adapter class
+# examples/redis_storage.rb
+class RedisStorage
+  # ... implementation
+end
+
+PromptManager::Prompt.storage_adapter = RedisStorage.new(Redis.new)
 ```
 
-##### model
-The `model` configuration parameter is the actual class name of the `ActiveRecord::Base` or `ApplicationRecord` (if you are using a rails application) that contains the content used for prompts.
+## Extensible Architecture
+
+PromptManager is designed to be extended:
+
+### Extension Points
 
-##### id_column
-The `id_column` contains the name of the column that contains the "prompt ID" content.  It can be either a `String` or `Symbol` value.
+1. **Storage Adapters** - Implement your own persistence layer
+2. **Directive Processors** - Add custom directives
+3. **Search Processors** - Integrate external search tools
+4. **Serializers** - Support different parameter formats
 
-##### text_column
-The `text_column` contains name of the column that contains the actual raw text of the prompt.  This raw text can include the keywords which will be replaced by values from the parameters Hash.  The column name value can be either a `String` or a `Symbol`.
+### Potential Extensions
 
-##### parameters_column
-The `parameters_column` contains the name of the column that contains the parameters used to replace keywords in the prompt text.  This column in the database model is expected to be serialized.  The `ActiveRecordAdapter` currently has a kludge bit of code that assumes that the serialization is done with JSON.  The value of the parameters_column can be either a `String` or a `Symbol`.
+- **CloudStorageAdapter** - S3, Google Cloud Storage
+- **RedisAdapter** - For caching and fast access
+- **ApiAdapter** - REST API backend
+- **GraphQLAdapter** - GraphQL endpoint storage
+- **GitAdapter** - Version controlled prompts
 
-TODO: fix the kludge so that any serialization can be used.
+## Roadmap
 
-### Other Potential Storage Adapters
+### v0.9.0 - Modern Prompt Format (Breaking Changes)
+- **Markdown Support**: Full `.md` file support with YAML front matter
+- **Modern Parameter Syntax**: Support for `{{keyword}}` format
+- **Enhanced API**: New `set_parameter()` and `get_parameter()` methods
+- **Parameter Validation**: Built-in validation based on specifications
+- **HTML Comments**: Support for `<!-- comments -->`
+- **Migration Tools**: Automated conversion utilities
 
-There are many possibilities to expand this plugin concept of the storage adapter.  Here are some for consideration:
+### v1.0.0 - Stability Release
+- Performance optimizations
+- Complete documentation
+- Production hardening
 
-- RedisAdapter - For caching prompts or temporary storage
-- ApiAdapter - Use some end-point to CRUD a prompt
-- CloudStorageAdapter - Store prompts in cloud storage services
+### Future Enhancements
+- Additional storage adapters
+- Enhanced directive system with plugins
+- Prompt versioning and inheritance
+- Performance optimizations for large collections
 
 ## Development