prompt_manager 0.4.1 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 91badda19121014eee4d6c28b1cb882347d951af971d9afb021b00e558cb3fdc
-  data.tar.gz: 500c401e8ced2be25adacaf374a6e18e73c42ec811e7bb632155de9d5626370c
+  metadata.gz: 267e48a65e377f2fc3390d3bbff69a6e1f241c96951fc86437f5ab02d2e93daa
+  data.tar.gz: db8af7d8515e72bf12f847e3544a7413a0acb0707a8321bcae7f13e1ffa1ff86
 SHA512:
-  metadata.gz: 89c274000ee45ddb047eb33ab232d0e300fab51be512b1181e6d9d9dbf6abe5213013229bfd0450e948b78b09aa00aab882131ed066c7529e27c5865b8251647
-  data.tar.gz: 4fc94add6f43ab7b0f289849e68c3c6c9f35ca24a90983a459cd2ea9043de03b8b2f9ed6701a3a614be7efecd1fc5557ab0fb0d252e84dc628d38f9f80d2d9ba
+  metadata.gz: 259cb1494a9390f94d40c669bd87960d9257a9126c3d71cbc6fdcdbaa82dae3e0209f23acc322d019e781abe48090a070fba20761a2ffe8a1818a303c8ef2741
+  data.tar.gz: 97e6ce3eb0743b02804b44c745a2710f15e4048957c7a039c6313b2d061d65660f9d4435465fd0139ff8e348ba841cb4ae543d06a7709391fab19a6419ea1e55

data/.irbrc

@@ -0,0 +1,14 @@
+require_relative 'lib/prompt_manager'
+require_relative 'lib/prompt_manager/storage/file_system_adapter'
+
+HERE = Pathname.new __dir__
+
+PromptManager::Storage::FileSystemAdapter.config do |config|
+  config.prompts_dir        = HERE + 'examples/prompts_dir'
+  # config.search_proc      = nil     # default
+  # config.prompt_extension = '.txt'  # default
+  # config.parms+_extension = '.json' # default
+end
+
+PromptManager::Prompt.storage_adapter = PromptManager::Storage::FileSystemAdapter.new
+

data/CHANGELOG.md

@@ -1,27 +1,41 @@
-## [0.4.1] = 2023-12-29
+## Unreleased
+
+## Released
+### [0.5.0] = 2025-03-29
+- Major refactoring of to improve processing of parameters and directives.
+- Added PromptManager::DirectiveProcessor as an example of how to implement custom directives.
+- Added support for //include directive that protects against loops.
+- Added support for embedding system environment variables.
+- Added support for ERB processing within a prompt.
+- Improved test coverage.
+
+### [0.4.2] = 2024-10-26
+- Added configurable parameter_regex to customize keyword pattern
+
+### [0.4.1] = 2023-12-29
 - Changed @directives from Hash to an Array
 - Fixed keywords not being substituted in directives
 
-## [0.4.0] = 2023-12-19
+### [0.4.0] = 2023-12-19
 - Add "//directives param(s)" with keywords just like the prompt text.
 
-## [0.3.3] = 2023-12-01
+### [0.3.3] = 2023-12-01
 - Added example of using the `search_proc` config parameter with the FileSystemAdapter.
 
-## [0.3.2] = 2023-12-01
+### [0.3.2] = 2023-12-01
 
 - The ActiveRecordAdapter is passing its unit tests
 - Dropped the concept of an sqlite3 adapter since active record can be used to access sqlite3 databases as well as the big boys.
 
-## [0.3.0] = 2023-11-28
+### [0.3.0] = 2023-11-28
 
 - **Breaking change** The value of the parameters Hash for a keyword is now an Array instead of a single value.  The last value in the Array is always the most recent value used for the given keyword.  This was done to support the use of a Readline::History object editing in the [aia](https://github.com/MadBomber/aia) CLI tool
 
-## [0.2.0] - 2023-11-21
+### [0.2.0] - 2023-11-21
 
 - **Breaking change to FileSystemAdapter config process**
 - added list and path as extra methods in FileSystemAdapter
 
-## [0.1.0] - 2023-11-16
+### [0.1.0] - 2023-11-16
 
 - Initial release using the FileSystemAdapter

data/README.md

@@ -9,6 +9,8 @@ Manage the parameterized prompts (text) used in generative AI (aka chatGPT, Open
 
 ## Table of Contents
 
+- [PromptManager](#promptmanager)
+  - [Table of Contents](#table-of-contents)
   - [Installation](#installation)
   - [Usage](#usage)
   - [Overview](#overview)
@@ -16,15 +18,15 @@ Manage the parameterized prompts (text) used in generative AI (aka chatGPT, Open
       - [What does a keyword look like?](#what-does-a-keyword-look-like)
       - [All about directives](#all-about-directives)
         - [Example Prompt with Directives](#example-prompt-with-directives)
-        - [Accessing Directives](#accessing-directives)
+        - [Accessing Directives and Setting Parameter Values](#accessing-directives-and-setting-parameter-values)
         - [Dynamic Directives](#dynamic-directives)
         - [Executing Directives](#executing-directives)
       - [Comments Are Ignored](#comments-are-ignored)
   - [Storage Adapters](#storage-adapters)
     - [FileSystemAdapter](#filesystemadapter)
       - [Configuration](#configuration)
-        - [prompts_dir](#prompts_dir)
-        - [search_proc](#search_proc)
+        - [prompts\_dir](#prompts_dir)
+        - [search\_proc](#search_proc)
         - [File Extensions](#file-extensions)
       - [Example Prompt Text File](#example-prompt-text-file)
       - [Example Prompt Parameters JSON File](#example-prompt-parameters-json-file)
@@ -32,15 +34,21 @@ Manage the parameterized prompts (text) used in generative AI (aka chatGPT, Open
     - [ActiveRecordAdapter](#activerecordadapter)
       - [Configuration](#configuration-1)
         - [model](#model)
-        - [id_column](#id_column)
-        - [text_column](#text_column)
-        - [parameters_column](#parameters_column)
+        - [id\_column](#id_column)
+        - [text\_column](#text_column)
+        - [parameters\_column](#parameters_column)
     - [Other Potential Storage Adapters](#other-potential-storage-adapters)
   - [Development](#development)
   - [Contributing](#contributing)
   - [License](#license)
 
 <!-- Tocer[finish]: Auto-generated, don't remove. -->
+### Latest Capabilities
+- **Directive Processing:** Processes directives such as `//include` (aliased as `//import`) with loop protection.
+- **ERB Processing:** Supports ERB templating within prompts.
+- **Environment Variable Embedding:** Automatically substitutes system environment variables in prompts.
+- **Improved Parameter Handling:** Refactored to maintain a history of parameter values.
+- **ActiveRecord Adapter:** Facilitates storing and retrieving prompts via an ActiveRecord model.
 
 ## Installation
 
@@ -60,25 +68,39 @@ See also [examples/using_search_proc.rb](examples/using_search_proc.rb)
 
 ## Overview
 
+### Prompt Initialization Options
+- `id`: A String name for the prompt.
+- `context`: An Array for additional context.
+- `directives_processor`: An instance of PromptManager::DirectiveProcessor (default), can be customized.
+- `external_binding`: A Ruby binding to be used for ERB processing.
+- `erb_flag`: Boolean flag to enable ERB processing in the prompt text.
+- `envar_flag`: Boolean flag to enable environment variable substitution in the prompt text.
+
 The `prompt_manager` gem provides functionality to manage prompts that have keywords and directives for use with generative AI processes.
 
 ### Generative AI (gen-AI)
 
-Gen-AI deals with the conversion (some would say execution) of a human natural language text (the "prompt") into somthing else using what are known as large language models (LLM) such as those available from OpenAI.  A parameterized prompt is one in which there are embedded keywords (parameters) which are place holders for other text to be inserted into the prompt.
+Gen-AI deals with the conversion (some would say execution) of a human natural language text (the "prompt") into something else using what are known as large language models (LLM) such as those available from OpenAI.  A parameterized prompt is one in which there are embedded keywords (parameters) which are place holders for other text to be inserted into the prompt.
 
 The prompt_manager uses a regular expression to identify these keywords within the prompt. It uses the keywords as keys in a `parameters` Hash which is stored with the prompt text in a serialized form - for example as JSON.
 
 #### What does a keyword look like?
 
-The current hard-coded REGEX for a [KEYWORD] identifies any all [UPPERCASE_TEXT] enclosed in square brackets as a keyword. [KEYWORDS CAN ALSO HAVE SPACES] as well as the underscore character.
+By default, any text matching `[UPPERCASE_TEXT]` enclosed in square brackets is treated as a keyword. [KEYWORDS CAN ALSO HAVE SPACES] as well as the underscore character.
+
+You can customize the keyword pattern by setting a different regular expression:
 
-This is just the initial convention adopted by prompt_manager. It is intended that this REGEX be configurable so that the prompt_manager can be used with other conventions.
+```ruby
+# Use {{param}} style instead of [PARAM]
+PromptManager::Prompt.parameter_regex = /(\{\{[A-Za-z_]+\}\})/
+```
 
+The regex must include capturing parentheses () to extract the keyword. The default regex is `/(\[[A-Z _|]+\])/`.
 #### All about directives
 
 A directive is a line in the prompt text that starts with the two characters '//' - slash slash - just like in the old days of IBM JCL - Job Control Language.  A prompt can have zero or more directives.  Directives can have parameters and can make use of keywords.
 
-The `prompt_manager` only collects directives.  It extracts keywords from directive lines and provides the substitution of those keywords with other text just like it does for the prompt.  
+The `prompt_manager` only collects directives.  It extracts keywords from directive lines and provides the substitution of those keywords with other text just like it does for the prompt.
 
 ##### Example Prompt with Directives
 
@@ -96,20 +118,30 @@ __END__
 Computers will never replace Frank Sinatra
 ```
 
-##### Accessing Directives
+##### Accessing Directives and Setting Parameter Values
 
-Getting directives from a prompt is as easy as getting the kewyords:
+Getting directives and keywords from a prompt is straightforward:
 
 ```ruby
-prompt = PromptManager::Prompt.new(...)
-prompt.keywords   #=> an Array
+prompt = PromptManager::Prompt.new(id: 'some_id')
+prompt.keywords   #=> an Array of keywords found in the prompt text
 prompt.directives #=> an Array of entries like: ['directive', 'parameters']
 
+# Or directly update the parameters hash
+prompt.parameters = {
+  "[KEYWORD1]" => "value1",
+  "[KEYWORD2]" => "value2"
+}
+
+# After setting parameter values, call to_s to build the final prompt
 # to_s builds the prompt by substituting
-# values for keywords amd removing comments.
+# values for keywords and removing comments.
 # The resulting text contains directives and
 # prompt text ready for the LLM process.
-puts prompt.to_s  
+final_text = prompt.to_s
+
+# To persist any parameter changes to storage
+prompt.save
 ```
 
 The entries in the Array returned by the `prompt.directives` method is in the order that the directives were defined within the prompt.  Each entry has two elements:
@@ -132,7 +164,7 @@ Since directies are collected after the keywords in the prompt have been substit
 
 The `prompt_manager` gem only collects directives.  Executing those directives is left up to some down stream process.  Here are some ideas on how directives could be used in prompt downstream process:
 
-- "//model gpt-5" could be used to set the LLM model to be used for a specific prompt.  
+- "//model gpt-5" could be used to set the LLM model to be used for a specific prompt.
 - "//backend mods" could be used to set the backend prompt processor on the command line to be the `mods` utility.
 - "//include path_to_file" could be used to add the contents of a file to the prompt.
 - "//chat" could be used to send the prompts and then start up a chat session about the prompt and its response.
@@ -166,8 +198,8 @@ Use a `config` block to establish the configuration for the class.
 
 ```ruby
 PromptManager::Storage::FileSystemAdapter.config do |o|
-  o.prompts_dir       = "path/to/prompts_directory" 
-  o.search_proc       = nil     # default 
+  o.prompts_dir       = "path/to/prompts_directory"
+  o.search_proc       = nil     # default
   o.prompt_extension  = '.txt'  # default
   o.params_extension  = '.json' # default
 end
@@ -177,7 +209,7 @@ The `config` block returns `self` so that means you can do this to setup the sto
 
 ```ruby
 PromptManager::Prompt
-  .storage_adapter = 
+  .storage_adapter =
     PromptManager::Storage::FileSystemAdapter
       .config do |config|
         config.prompts_dir = 'path/to/prompts_dir'
@@ -192,7 +224,7 @@ An `ArgumentError` will be raised when `prompts_dir` does not exist or if it is
 
 ##### search_proc
 
-The default for `search_proc` is nil which means that the search will be preformed by a default `search` method which is basically reading all the prompt files to see which ones contain the search term. It will return an Array of prompt IDs for each prompt file found that contains the search term.  Its up to the application to select which returned prompt ID to use. 
+The default for `search_proc` is nil which means that the search will be preformed by a default `search` method which is basically reading all the prompt files to see which ones contain the search term. It will return an Array of prompt IDs for each prompt file found that contains the search term.  Its up to the application to select which returned prompt ID to use.
 
 There are faster ways to search and select files.  For example there are specialized search and selection utilities that are available for the command line. The `examples` directory contains a `bash` script named `rgfzf` that uses `rg` (aka `ripgrep`) to do the searching and `fzf` to do the selecting.
 
@@ -267,7 +299,7 @@ The `PromptManager::Prompt` class expects an instance of a storage adapter class
 
 ```ruby
 PromptManager::Prompt
-  .storage_adapter = 
+  .storage_adapter =
     PromptManager::Storage::ActiveRecordAdapter.config do |config|
       config.model              = DbPromptModel # any ActiveRecord::Base model
       config.id_column          = :prompt_name

data/Rakefile

@@ -8,7 +8,6 @@ end
 
 Tocer::Rake::Register.call
 
-
 require "bundler/gem_tasks"
 require "rake/testtask"
 

data/examples/directives.rb

@@ -0,0 +1,98 @@
+#!/usr/bin/env ruby
+# encoding: utf-8
+# frozen_string_literal: true
+# warn_indent: true
+##########################################################
+###
+##  File: directives.rb
+##  Desc: Demo of the PromptManager and FileStorageAdapter
+##  By:   Dewayne VanHoozer (dvanhoozer@gmail.com)
+##
+#
+
+param1 = "param_one"
+param2 = "param_two"
+
+
+class MyDirectives
+  def self.good_directive(*args)
+    puts "inside #{__method__} with these parameters:"
+    puts args.join(",\n")
+  end
+end
+
+def concept_break = print "\n------------------------\n\n\n"
+
+require_relative '../lib/prompt_manager'
+require_relative '../lib/prompt_manager/storage/file_system_adapter'
+
+require 'amazing_print'
+require 'pathname'
+
+require 'debug_me'
+include DebugMe
+
+HERE        = Pathname.new( __dir__ )
+PROMPTS_DIR = HERE + "prompts_dir"
+
+
+######################################################
+# Main
+
+at_exit do
+  puts
+  puts "Done."
+  puts
+end
+
+# Configure the Storage Adapter to use
+PromptManager::Storage::FileSystemAdapter.config do |config|
+  config.prompts_dir        = PROMPTS_DIR
+  # config.search_proc      = nil     # default
+  # config.prompt_extension = '.txt'  # default
+  # config.parms+_extension = '.json' # default
+end
+
+PromptManager::Prompt.storage_adapter = PromptManager::Storage::FileSystemAdapter.new
+
+# Use {parameter name} brackets to define a parameter
+PromptManager::Prompt.parameter_regex =  /\{[A-Za-z _|]+\}/ 
+
+# Retrieve a prompt
+prompt = PromptManager::Prompt.get(id: 'directive_example')
+
+# Shows prompt without comments or directives
+# It still has its parameter placeholders
+puts prompt
+concept_break
+
+puts "Directives in the prompt:"
+ap prompt.directives
+
+puts "Processing directives ..."
+prompt.directives.each do |entry|
+  if MyDirectives.respond_to? entry.first.to_sym
+    ruby = "MyDirectives.#{entry.first}(#{entry.last.gsub(' ', ',')})"
+    eval "#{ruby}"
+  else
+    puts "ERROR: there is no method: #{entry.first}"
+  end
+end
+
+concept_break
+
+
+
+puts "Parameters in the prompt:"
+ap prompt.parameters
+puts "-"*16
+
+puts "keywords:"
+ap prompt.keywords
+concept_break
+
+prompt.parameters['{language}'] << 'French'
+
+puts "After Substitution"
+puts prompt
+

data/examples/using_search_proc.rb

@@ -10,9 +10,6 @@
 ##
 #
 
-require 'debug_me'
-include DebugMe
-
 require 'prompt_manager'
 require 'prompt_manager/storage/file_system_adapter'
 

data/lib/prompt_manager/directive_processor.rb

@@ -0,0 +1,47 @@
+# lib/prompt_manager/directive_processor.rb
+
+# This is an example of a directive processor class.
+# It only supports the //include directive which is also
+# aliased as //import.
+
+module PromptManager
+  class DirectiveProcessor
+    EXCLUDED_METHODS = %w[ run initialize ]
+
+    def initialize
+      @prefix_size    = PromptManager::Prompt::DIRECTIVE_SIGNAL.size
+      @included_files = []
+    end
+
+    def run(directives)
+      return {} if directives.nil? || directives.empty?
+      directives.each do |key, _|
+        sans_prefix = key[@prefix_size..]
+        args        = sans_prefix.split(' ')
+        method_name = args.shift
+
+        if EXCLUDED_METHODS.include?(method_name)
+          directives[key] = "Error: #{method_name} is not a valid directive: #{key}"
+        elsif respond_to?(method_name)
+          directives[key] =  send(method_name, *args)
+        else
+          directives[key] = "Error: Unknown directive '#{key}'"
+        end
+      end
+      directives
+    end
+
+    def include(file_path)
+      if  File.exist?(file_path) &&
+          File.readable?(file_path) &&
+          !@included_files.include?(file_path)
+        content = File.read(file_path)
+        @included_files << file_path
+        content
+      else
+        "Error: File '#{file_path}' not accessible"
+      end
+    end
+    alias_method :import, :include
+  end
+end