prompt_manager 0.4.0 → 0.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 60ae8d4c5f669935906248aa0e5ffc628e6ef0fbcc28eb31c889eb85b200d370
-  data.tar.gz: 44e51b38634667930b5524232cefd63fe0dab3ef334500714361680009186bfd
+  metadata.gz: 6c566743ba6741a0787db6ea680e77676ab18282013e3475cad1c30b85b9ce1b
+  data.tar.gz: 1e9e57eecaf97d05799d6b173b1ce90518dc02abeeddbd50eed90c34c376509f
 SHA512:
-  metadata.gz: 634cc2d1cfdedaf1ca5f5a232334b7e8ccbad25dbb48e669daddf831e43eb45f03367f2921561cdb8ddcf1675ea48f1dcf8cc40defb479aa1eced3bb493442f9
-  data.tar.gz: f04cc92a7ac3c286fb3958eaa44358c402899a121e89ca41aa7bddd30c41bd704bed55dd57775340c622671d0fbf36eef48171ffc3d290f220d6306622486ef3
+  metadata.gz: e1bcda6797d895b4dd78baf6c6c6cd2b83e2f06bb3446f9112795a6d216eb9450af9f1c2d0b3f762d4f21ee40a1257fec8610eadec33f7f19dabf078e7a859bf
+  data.tar.gz: f69ed112a2b22801dc830d87b11f0a231755141fa4eccded5ee2dc8019343cf6cf61705af560fed211866869940029bbde88cca7c821423da245593839515e03

data/.irbrc

@@ -0,0 +1,14 @@
+require_relative 'lib/prompt_manager'
+require_relative 'lib/prompt_manager/storage/file_system_adapter'
+
+HERE = Pathname.new __dir__
+
+PromptManager::Storage::FileSystemAdapter.config do |config|
+  config.prompts_dir        = HERE + 'examples/prompts_dir'
+  # config.search_proc      = nil     # default
+  # config.prompt_extension = '.txt'  # default
+  # config.parms+_extension = '.json' # default
+end
+
+PromptManager::Prompt.storage_adapter = PromptManager::Storage::FileSystemAdapter.new
+

data/CHANGELOG.md

@@ -1,23 +1,34 @@
-## [0.4.0] = 2023-12-19
+## Unreleased
+
+## Released
+
+### [0.4.2] = 2024-10-26
+- Added configurable parameter_regex to customize keyword pattern
+
+### [0.4.1] = 2023-12-29
+- Changed @directives from Hash to an Array
+- Fixed keywords not being substituted in directives
+
+### [0.4.0] = 2023-12-19
 - Add "//directives param(s)" with keywords just like the prompt text.
 
-## [0.3.3] = 2023-12-01
+### [0.3.3] = 2023-12-01
 - Added example of using the `search_proc` config parameter with the FileSystemAdapter.
 
-## [0.3.2] = 2023-12-01
+### [0.3.2] = 2023-12-01
 
 - The ActiveRecordAdapter is passing its unit tests
 - Dropped the concept of an sqlite3 adapter since active record can be used to access sqlite3 databases as well as the big boys.
 
-## [0.3.0] = 2023-11-28
+### [0.3.0] = 2023-11-28
 
 - **Breaking change** The value of the parameters Hash for a keyword is now an Array instead of a single value.  The last value in the Array is always the most recent value used for the given keyword.  This was done to support the use of a Readline::History object editing in the [aia](https://github.com/MadBomber/aia) CLI tool
 
-## [0.2.0] - 2023-11-21
+### [0.2.0] - 2023-11-21
 
 - **Breaking change to FileSystemAdapter config process**
 - added list and path as extra methods in FileSystemAdapter
 
-## [0.1.0] - 2023-11-16
+### [0.1.0] - 2023-11-16
 
 - Initial release using the FileSystemAdapter

data/README.md

@@ -15,6 +15,10 @@ Manage the parameterized prompts (text) used in generative AI (aka chatGPT, Open
     - [Generative AI (gen-AI)](#generative-ai-gen-ai)
       - [What does a keyword look like?](#what-does-a-keyword-look-like)
       - [All about directives](#all-about-directives)
+        - [Example Prompt with Directives](#example-prompt-with-directives)
+        - [Accessing Directives](#accessing-directives)
+        - [Dynamic Directives](#dynamic-directives)
+        - [Executing Directives](#executing-directives)
       - [Comments Are Ignored](#comments-are-ignored)
   - [Storage Adapters](#storage-adapters)
     - [FileSystemAdapter](#filesystemadapter)
@@ -66,17 +70,25 @@ The prompt_manager uses a regular expression to identify these keywords within t
 
 #### What does a keyword look like?
 
-The current hard-coded REGEX for a [KEYWORD] identifies any all [UPPERCASE_TEXT] enclosed in square brackets as a keyword. [KEYWORDS CAN ALSO HAVE SPACES] as well as the underscore character.
+By default, any text matching `[UPPERCASE_TEXT]` enclosed in square brackets is treated as a keyword. [KEYWORDS CAN ALSO HAVE SPACES] as well as the underscore character.
 
-This is just the initial convention adopted by prompt_manager. It is intended that this REGEX be configurable so that the prompt_manager can be used with other conventions.
+You can customize the keyword pattern by setting a different regular expression:
 
+```ruby
+# Use {{param}} style instead of [PARAM]
+PromptManager::Prompt.parameter_regex = /(\{\{[A-Za-z_]+\}\})/
+```
+
+The regex must include capturing parentheses () to extract the keyword. The default regex is `/(\[[A-Z _|]+\])/`.
 #### All about directives
 
 A directive is a line in the prompt text that starts with the two characters '//' - slash slash - just like in the old days of IBM JCL - Job Control Language.  A prompt can have zero or more directives.  Directives can have parameters and can make use of keywords.
 
-The `prompt_manager` only collections directives.  It extracts keywords from directive lines and provides the substitution of those keywords with other text just like it does for the prompt.  Remember a directive is part of the prompt.
+The `prompt_manager` only collects directives.  It extracts keywords from directive lines and provides the substitution of those keywords with other text just like it does for the prompt.  
+
+##### Example Prompt with Directives
 
-Here is an example problem with comments, directives and keywords:
+Here is an example prompt text file with comments, directives and keywords:
 
 ```
 # prompts/sing_a_song.txt
@@ -90,12 +102,14 @@ __END__
 Computers will never replace Frank Sinatra
 ```
 
+##### Accessing Directives
+
 Getting directives from a prompt is as easy as getting the kewyords:
 
 ```ruby
 prompt = PromptManager::Prompt.new(...)
 prompt.keywords   #=> an Array
-prompt.directives #=> a Hash
+prompt.directives #=> an Array of entries like: ['directive', 'parameters']
 
 # to_s builds the prompt by substituting
 # values for keywords amd removing comments.
@@ -104,9 +118,33 @@ prompt.directives #=> a Hash
 puts prompt.to_s  
 ```
 
-The Hash returned by the `prompt.directives` method has as its key the directive name (without the double-slash).  Its also case-sensitive.  So any typo made in editing the prompt with regard to the case of the directive's name will impact the backend processing of the prompt.
+The entries in the Array returned by the `prompt.directives` method is in the order that the directives were defined within the prompt.  Each entry has two elements:
+
+- directive name (without the // characters)
+- parameter string for the directive
+
+##### Dynamic Directives
+
+Since directies are collected after the keywords in the prompt have been substituted for their values, it is possible to have dynamically generated directives as part of a prompt.  For example:
+
+```
+//[COMMAND] [OPTIONS]
+# or
+[SOMETHING]
+```
+... where [COMMAND] gets replaced by some directive name.  [SOMETHING] could be replaced by "//directive options"
+
+##### Executing Directives
+
+The `prompt_manager` gem only collects directives.  Executing those directives is left up to some down stream process.  Here are some ideas on how directives could be used in prompt downstream process:
+
+- "//model gpt-5" could be used to set the LLM model to be used for a specific prompt.  
+- "//backend mods" could be used to set the backend prompt processor on the command line to be the `mods` utility.
+- "//include path_to_file" could be used to add the contents of a file to the prompt.
+- "//chat" could be used to send the prompts and then start up a chat session about the prompt and its response.
+
+Its all up to how your application wants to support directives or not.
 
-The value for each key is a String exactly as entered in the prompt file.
 
 #### Comments Are Ignored
 

data/Rakefile

@@ -8,7 +8,6 @@ end
 
 Tocer::Rake::Register.call
 
-
 require "bundler/gem_tasks"
 require "rake/testtask"
 

data/examples/directives.rb

@@ -0,0 +1,98 @@
+#!/usr/bin/env ruby
+# encoding: utf-8
+# frozen_string_literal: true
+# warn_indent: true
+##########################################################
+###
+##  File: directives.rb
+##  Desc: Demo of the PromptManager and FileStorageAdapter
+##  By:   Dewayne VanHoozer (dvanhoozer@gmail.com)
+##
+#
+
+param1 = "param_one"
+param2 = "param_two"
+
+
+class MyDirectives
+  def self.good_directive(*args)
+    puts "inside #{__method__} with these parameters:"
+    puts args.join(",\n")
+  end
+end
+
+def concept_break = print "\n------------------------\n\n\n"
+
+require_relative '../lib/prompt_manager'
+require_relative '../lib/prompt_manager/storage/file_system_adapter'
+
+require 'amazing_print'
+require 'pathname'
+
+require 'debug_me'
+include DebugMe
+
+HERE        = Pathname.new( __dir__ )
+PROMPTS_DIR = HERE + "prompts_dir"
+
+
+######################################################
+# Main
+
+at_exit do
+  puts
+  puts "Done."
+  puts
+end
+
+# Configure the Storage Adapter to use
+PromptManager::Storage::FileSystemAdapter.config do |config|
+  config.prompts_dir        = PROMPTS_DIR
+  # config.search_proc      = nil     # default
+  # config.prompt_extension = '.txt'  # default
+  # config.parms+_extension = '.json' # default
+end
+
+PromptManager::Prompt.storage_adapter = PromptManager::Storage::FileSystemAdapter.new
+
+# Use {parameter name} brackets to define a parameter
+PromptManager::Prompt.parameter_regex =  /\{[A-Za-z _|]+\}/ 
+
+# Retrieve a prompt
+prompt = PromptManager::Prompt.get(id: 'directive_example')
+
+# Shows prompt without comments or directives
+# It still has its parameter placeholders
+puts prompt
+concept_break
+
+puts "Directives in the prompt:"
+ap prompt.directives
+
+puts "Processing directives ..."
+prompt.directives.each do |entry|
+  if MyDirectives.respond_to? entry.first.to_sym
+    ruby = "MyDirectives.#{entry.first}(#{entry.last.gsub(' ', ',')})"
+    eval "#{ruby}"
+  else
+    puts "ERROR: there is no method: #{entry.first}"
+  end
+end
+
+concept_break
+
+
+
+puts "Parameters in the prompt:"
+ap prompt.parameters
+puts "-"*16
+
+puts "keywords:"
+ap prompt.keywords
+concept_break
+
+prompt.parameters['{language}'] << 'French'
+
+puts "After Substitution"
+puts prompt
+

data/examples/prompts_dir/directive_example.txt

@@ -0,0 +1,14 @@
+# directive_example.txt
+# Desc: Shows how directives work
+
+//good_directive  param1 param2
+//bad_directive   param3 param4
+
+say hello to me in {language} as well as English.
+
+# The default parameter is delimited by square brackets
+# and is all uppercase
+
+write a [PROGRAMMING LANGUAGE] program that predicts the lottery.
+
+

data/examples/using_search_proc.rb

@@ -10,9 +10,6 @@
 ##
 #
 
-require 'debug_me'
-include DebugMe
-
 require 'prompt_manager'
 require 'prompt_manager/storage/file_system_adapter'
 

data/lib/prompt_manager/prompt.rb

@@ -6,23 +6,27 @@
 # comment removal. It communicates with a storage system through a storage 
 # adapter.
 #
-# Directives can be anything required by the backend
-# prompt processing functionality.  Directives are
-# available along with the prompt text w/o comments.
-# Keywords can be used in the directives.
+# Directives are collected into an Array where each entry is an Array
+# of two elements.  The first is the directive name as a String. The 
+# second is a string of parameters used by the directive.
+# 
+# Directives are collected from the prompt after keyword
+# substitution has occured.  This means that directives within a
+# prompt can be dynamic.
 #
-# It is expected that the backend prompt processes will
-# act on and remove the directives before passing the
-# prompt text on through the LLM.
+# PromptManager does not execute directives.  They
+# are made available to be passed on to down stream
+# process.
 
 class PromptManager::Prompt
   COMMENT_SIGNAL    = '#'   # lines beginning with this are a comment
   DIRECTIVE_SIGNAL  = '//'  # Like the old IBM JCL
-  PARAMETER_REGEX   = /(\[[A-Z _|]+\])/
+  DEFAULT_PARAMETER_REGEX = /(\[[A-Z _|]+\])/
   @storage_adapter  = nil
+  @parameter_regex  = DEFAULT_PARAMETER_REGEX
 
   class << self
-    attr_accessor :storage_adapter
+    attr_accessor :storage_adapter, :parameter_regex
 
     alias_method :get, :new
 
@@ -58,7 +62,7 @@ class PromptManager::Prompt
   
   # SMELL:  Does the db (aka storage adapter) really need
   #         to be accessible by the main program?
-  attr_accessor :db, :id, :text, :parameters
+  attr_accessor :db, :id, :text, :parameters, :directives
 
 
   # Retrieve the specific prompt ID from the Storage system.
@@ -76,10 +80,9 @@ class PromptManager::Prompt
     @text       = @record[:text]
     @parameters = @record[:parameters]
     @keywords   = []  # Array of String
-    @directives = {}  # Hash with directive as key, parameters as value
+    @directives = []  # Array of arrays. directive is first entry, rest are parameters
 
     update_keywords
-    update_directives
 
     build
   end
@@ -121,11 +124,12 @@ class PromptManager::Prompt
   # the comments.
   #  
   def build
-    @prompt = text.gsub(PARAMETER_REGEX) do |match|
+    @prompt = text.gsub(self.class.parameter_regex) do |match|
                 param_name = match
                 Array(parameters[param_name]).last || match
               end
-
+              
+    save_directives(@prompt)
     remove_comments
   end
 
@@ -135,16 +139,11 @@ class PromptManager::Prompt
   end
 
 
-  def directives
-    update_directives    
-  end
-
-
   ######################################
   private
 
   def update_keywords
-    @keywords = @text.scan(PARAMETER_REGEX).flatten.uniq
+    @keywords = @text.scan(self.class.parameter_regex).flatten.uniq
     @keywords.each do |kw|
       @parameters[kw] = [] unless @parameters.has_key?(kw)
     end
@@ -153,14 +152,16 @@ class PromptManager::Prompt
   end
 
 
-  def update_directives
-    @text.split("\n").each do |a_line|
+  def save_directives(keyword_substituted_string)
+    @directives = []
+
+    keyword_substituted_string.split("\n").each do |a_line|
       line = a_line.strip
       next unless line.start_with?(DIRECTIVE_SIGNAL)
       
-      parts     = line.split(' ')
-      directive = parts.shift()[DIRECTIVE_SIGNAL.length..] # drop the directive signal
-      @directives[directive] = parts.join(' ')
+      parts       = line.split(' ')
+      directive   = parts.shift[DIRECTIVE_SIGNAL.length..] # drop the directive signal
+      @directives << [directive, parts.join(' ')]
     end
 
     @directives
@@ -171,7 +172,8 @@ class PromptManager::Prompt
     lines           = @prompt
                         .split("\n")
                         .reject{|a_line| 
-                          a_line.strip.start_with?(COMMENT_SIGNAL)
+                          a_line.strip.start_with?(COMMENT_SIGNAL)  ||
+                          a_line.strip.start_with?(DIRECTIVE_SIGNAL)
                         }
 
     # Remove empty lines at the start of the prompt

data/lib/prompt_manager/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module PromptManager
-  VERSION = "0.4.0"
+  VERSION = "0.4.2"
 end

data/lib/prompt_manager.rb

@@ -2,6 +2,8 @@
 #
 # frozen_string_literal: true
 
+require 'ostruct'
+
 require_relative "prompt_manager/version"
 require_relative "prompt_manager/storage"
 require_relative "prompt_manager/prompt"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: prompt_manager
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.4.2
 platform: ruby
 authors:
 - Dewayne VanHoozer
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-12-19 00:00:00.000000000 Z
+date: 2024-10-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -66,6 +66,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: ostruct
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: tocer
   requirement: !ruby/object:Gem::Requirement
@@ -80,8 +94,36 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: simplecov
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: "Manage the parameterized prompts (text) used in generative AI (aka chatGPT,
-  \nOpenAI, et.al.) using storage adapters such as FileSystemAdapter, \nSqliteAdapter
+  \nOpenAI, et.al.) using storage adapters such as FileSystemAdapter, \nSqliteAdapter,
   and ActiveRecordAdapter.\n"
 email:
 - dvanhoozer@gmail.com
@@ -90,11 +132,14 @@ extensions: []
 extra_rdoc_files: []
 files:
 - ".envrc"
+- ".irbrc"
 - CHANGELOG.md
 - LICENSE
 - LICENSE.txt
 - README.md
 - Rakefile
+- examples/directives.rb
+- examples/prompts_dir/directive_example.txt
 - examples/prompts_dir/todo.json
 - examples/prompts_dir/todo.txt
 - examples/prompts_dir/toy/8-ball.txt
@@ -130,7 +175,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.1
+rubygems_version: 3.5.22
 signing_key:
 specification_version: 4
 summary: Manage prompts for use with gen-AI processes