foobara-llm-backed-command 0.0.4 → 0.0.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 932878d2d01d4d804a4aca75cf53ccbbff901cf2e54067d52a781298d3ebea2a
-  data.tar.gz: 96c9c8a664b7fa493d740a222b5429d2259af317131c2ef4ebe23be6a730a68f
+  metadata.gz: b425fcdb4e888c59b2bc32f4a7c4a4d066b11cf6fcfee6f6dcb8fc00da88cc21
+  data.tar.gz: b27ebd00c36a49a2e79001e1122e783a4dcd1406a4d5ef61891af726c64821df
 SHA512:
-  metadata.gz: 947393bec6c00defe7198ea96f69c34a90f27d2beb9c35531e39b09c436c172b243a93e1be49b38cb64ecaf73f8a5ee572868a3cf7de89f343b39041ab47d650
-  data.tar.gz: f51fb84a77878d535c28699879c8ee7bc4dd643eaffbb3269aa679f994170ec0fc56c99bcb55c4c7f8570f95152bc8a12440860fcc552216d10d435ce8cad861
+  metadata.gz: 24facf55a6108124f8f0109b53acd0e2b5782d52f4ee812f8a7f223f725cdaedbb79a3bad11b0c67efa0253c612edacf50f7db56025c91991dac23e7344306f7
+  data.tar.gz: 80d4bd1e19ecdfd28b58ff84026ed5245c06dce067a0a7dffb5ed887bf97fe35395274eb2f25a581ab08ba53a7448b9e01dcaf67c032ef07e70c4096cd89994f

data/CHANGELOG.md

@@ -1,6 +1,11 @@
-## [0.0.4] - 2025-03-06
+## [0.0.6] - 2025-05-21
 
-- Defer to downstream default model if none specifiedz
+- Give an error if no ai api services have been provided
+
+## [0.0.5] - 2025-03-06
+
+- Defer to downstream default model if none specified
+- Split LlmBackedCommand into a Command and a LlmBackedExecuteMethod mixin
 
 ## [0.0.3] - 2025-03-05
 

data/README.md

@@ -1,6 +1,22 @@
-# Foobara::LlmBackedCommand
+# Foobara::LlmBackedCommand/Foobara::LlmBackedExecuteMethod
 
-Provides a clean and quick way to implement a Foobara::Command that has logic backed by an LLM.
+Provides a clean and quick way to implement a Foobara::Command that defers to an LLM for the answer
+
+<!-- TOC -->
+* [Foobara::LlmBackedCommand/Foobara::LlmBackedExecuteMethod](#foobarallmbackedcommandfoobarallmbackedexecutemethod)
+  * [Installation](#installation)
+  * [Usage](#usage)
+    * [Choosing a different model and llm service](#choosing-a-different-model-and-llm-service)
+    * [Using it as a mixin instead of a class](#using-it-as-a-mixin-instead-of-a-class)
+    * [Typical Foobara stuff: exposing it on the command-line](#typical-foobara-stuff-exposing-it-on-the-command-line)
+    * [Exposing commands through Rack](#exposing-commands-through-rack)
+    * [Exposing commands through Rails](#exposing-commands-through-rails)
+    * [Use with models](#use-with-models)
+    * [Use with entities](#use-with-entities)
+    * [Complete scripts to play with](#complete-scripts-to-play-with)
+  * [Contributing](#contributing)
+  * [License](#license)
+<!-- TOC -->
 
 ## Installation
 
@@ -9,12 +25,471 @@ Typical stuff: add `gem "foobara-llm-backed-command` to your Gemfile or .gemspec
 
 ## Usage
 
-TODO: Write usage instructions here
+To play with these examples you can do `gem install foobara-llm-backed-command foobara-anthropic-api`
+and then the following:
+
+```ruby
+ENV["ANTHROPIC_API_KEY"] = "<your key here>"
+
+require 'foobara/llm_backed_command'
+
+class DetermineLanguage < Foobara::LlmBackedCommand
+  inputs code_snippet: :string
+  result most_likely: :string, probabilities: { ruby: :float, c: :float, smalltalk: :float, java: :float }
+end
+
+puts DetermineLanguage.run!(code_snippet: "puts 'Hello, World'")
+```
+
+Running this script outputs:
+
+```
+$ ./demo
+{most_likely: "ruby", probabilities: {ruby: 0.95, c: 0.01, smalltalk: 0.02, java: 0.02}}
+```
+
+Here we have only specified the command name and the inputs/result. Our LLM of choice
+will be used to get the answer, and it will be structured according to the result type.
+
+The default LLM model is claude-3-7-sonnet, but you can use others:
+
+### Choosing a different model and llm service
+
+One way to do this is by creating an input called `llm_model`
+
+```ruby
+ENV["ANTHROPIC_API_KEY"] = "<your key here>"
+ENV["OPENAI_API_KEY"] = "<your key here>"
+ENV["OLLAMA_API_URL"] = "<your ollama API if different than http://localhost:11434>"
+
+require "foobara/anthropic_api"
+require "foobara/open_ai_api"
+require "foobara/ollama_api"
+require 'foobara/llm_backed_command'
+
+class DetermineLanguage < Foobara::LlmBackedCommand
+  inputs do
+    code_snippet :string, :required
+    llm_model Foobara::Ai::AnswerBot::Types.model_enum, default: "claude-3-7-sonnet-20250219"
+  end
+
+  result most_likely: :string, probabilities: { ruby: :float, c: :float, smalltalk: :float, java: :float }
+end
+
+inputs = { llm_model: "chat-gpt-3-5-turbo", code_snippet: "puts 'Hello, World'" }
+command = DetermineLanguage.new(inputs)
+outcome = command.run
+
+puts outcome.success? ? outcome.result : outcome.errors_hash
+```
+
+### Using it as a mixin instead of a class
+
+If you need the inheritance slot for some other command base class, you can use the mixin:
+
+```ruby
+class DetermineLanguage < Foobara::Command
+  include Foobara::LlmBackedExecuteMethod
+
+  inputs code_snippet: :string
+  result most_likely: :string, probabilities: { ruby: :float, c: :float, smalltalk: :float, java: :float }
+end
+```
+
+### Typical Foobara stuff: exposing it on the command-line
+
+Probably best to refer to Foobara for details instead of turning this README.md into a Foobara tutorial, but
+these can be used as any other Foobara command. So exposing
+the command on the command line is easy (requires `gem install foobara-sh-cli-connector fooara-dotenv-loader`)
+(switching to dotenv for convenience):
+
+```ruby
+#!/usr/bin/env ruby
+
+require "foobara/load_dotenv"
+Foobara::LoadDotenv.run!(dir: __dir__)
+
+require "foobara/anthropic_api"
+require "foobara/open_ai_api"
+require "foobara/ollama_api"
+require "foobara/llm_backed_command"
+require "foobara/sh_cli_connector"
+
+class DetermineLanguage < Foobara::LlmBackedCommand
+  inputs do
+    code_snippet :string, :required
+    llm_model Foobara::Ai::AnswerBot::Types.model_enum, default: "claude-3-7-sonnet-20250219"
+  end
+  result most_likely: :string, probabilities: { ruby: :float, c: :float, smalltalk: :float, java: :float }
+end
+
+Foobara::CommandConnectors::ShCliConnector.new(single_command_mode: DetermineLanguage).run(ARGV)
+```
+
+which allows:
+
+```
+$ ./determine-language --help
+Usage: determine-language [INPUTS]
+
+Inputs:
+
+ -c, --code-snippet CODE_SNIPPET  Required
+ -l, --llm-model LLM_MODEL        One of: babbage-002, chatgpt-4o-latest, claude-2.0, claude-2.1,
+                                    claude-3-5-haiku-20241022, claude-3-5-sonnet-20240620, claude-3-5-sonnet-20241022,
+                                    claude-3-7-sonnet-20250219, claude-3-haiku-20240307, claude-3-opus-20240229,
+                                    claude-3-sonnet-20240229, dall-e-2, dall-e-3, davinci-002, gpt-3.5-turbo,
+                                    gpt-3.5-turbo-0125, gpt-3.5-turbo-1106, gpt-3.5-turbo-16k, gpt-3.5-turbo-instruct,
+                                    gpt-3.5-turbo-instruct-0914, gpt-4, gpt-4-0125-preview, gpt-4-0613,
+                                    gpt-4-1106-preview, gpt-4-turbo, gpt-4-turbo-2024-04-09, gpt-4-turbo-preview,
+                                    gpt-4.5-preview, gpt-4.5-preview-2025-02-27, gpt-4o, gpt-4o-2024-05-13,
+                                    gpt-4o-2024-08-06, gpt-4o-2024-11-20, gpt-4o-audio-preview, gpt-4o-audio-preview-2024-10-01,
+                                    gpt-4o-audio-preview-2024-12-17, gpt-4o-mini, gpt-4o-mini-2024-07-18,
+                                    gpt-4o-mini-audio-preview, gpt-4o-mini-audio-preview-2024-12-17,
+                                    gpt-4o-mini-realtime-preview, gpt-4o-mini-realtime-preview-2024-12-17,
+                                    gpt-4o-mini-search-preview, gpt-4o-mini-search-preview-2025-03-11,
+                                    gpt-4o-realtime-preview, gpt-4o-realtime-preview-2024-10-01, gpt-4o-realtime-preview-2024-12-17,
+                                    gpt-4o-search-preview, gpt-4o-search-preview-2025-03-11, llama3:8b,
+                                    o1, o1-2024-12-17, o1-mini, o1-mini-2024-09-12, o1-preview, o1-preview-2024-09-12,
+                                    o3-mini, o3-mini-2025-01-31, omni-moderation-2024-09-26, omni-moderation-latest,
+                                    smollm2:135m, text-embedding-3-large, text-embedding-3-small, text-embedding-ada-002,
+                                    tts-1, tts-1-1106, tts-1-hd, tts-1-hd-1106, whisper-1.
+                                  Default: "claude-3-7-sonnet-20250219"
+```
+
+Running the program:
+
+```
+$ ./determine-language --code-snippet "Transcript show: 'Hello, World'"
+most_likely: "smalltalk",
+probabilities: {
+  ruby: 0.1,
+  c: 0.05,
+  smalltalk: 0.8,
+  java: 0.05
+}
+```
+
+### Exposing commands through Rack
+
+Or you can spin up a quick json API either through Rack or the Rails router. Here's an example with the rack connector
+(requires `gem install foobara-rack-controller rackup puma`):
+
+```ruby
+#!/usr/bin/env ruby
+
+require "foobara/load_dotenv"
+Foobara::LoadDotenv.run!(dir: __dir__)
+
+require "foobara/llm_backed_command"
+require "foobara/rack_connector"
+require "rackup/server"
+
+class DetermineLanguage < Foobara::LlmBackedCommand
+  inputs code_snippet: :string
+  result probabilities: { ruby: :float, c: :float, smalltalk: :float, java: :float },
+         most_likely: :string
+end
+
+command_connector = Foobara::CommandConnectors::Http::Rack.new
+command_connector.connect(DetermineLanguage)
+
+Rackup::Server.start(app: command_connector)
+```
+
+After we run this script we can hit it with curl:
+
+```
+$ curl http://localhost:9292/run/DetermineLanguage?code_snippet=System.out.println
+{"probabilities":{"ruby":0.05,"c":0.1,"smalltalk":0.05,"java":0.8},"most_likely":"java"}
+```
+
+And we can run it from other systems in either Ruby or TypeScript.
+
+In Ruby (requires `gem install foobara-remote-imports`):
+
+```ruby
+#!/usr/bin/env ruby
+
+require "foobara/remote_imports"
+
+Foobara::RemoteImports::ImportCommand.run!(manifest_url: "http://localhost:9292/manifest", cache: true)
+
+puts DetermineLanguage.run!(code_snippet: "System.out.println")
+```
+
+This outputs:
+
+```
+$ ./determine-language-client
+{probabilities: {ruby: 0.05, c: 0.05, smalltalk: 0.1, java: 0.8}, most_likely: "java"}
+```
+
+And we can also generate a remote command in TypeScript (requires `gem install foob`)
+
+From inside a TypeScript project:
+
+```
+$ foob g typescript-remote-commands --manifest-url http://localhost:9292/manifest
+```
+
+This will generate code so that we can do:
+
+```TypeScript
+import { DetermineLanguage } from "./DetermineLanguage"
+
+const command = new DetermineLanguage({code_snippet: "System.out.println"})
+const outcome = await command.run()
+
+if (outcome.isSuccess) {
+  console.log(outcome.result)
+} else {
+  console.error(outcome.errors_hash)
+}
+```
+
+and everything will be fully typed, inputs, result, etc.
+
+### Exposing commands through Rails
+
+This has become too much of a Foobara tutorial so instead please refer to
+https://github.com/foobara/rails-command-connector
+
+### Use with models
+
+You can of course use this with whatever Foobara concepts you want, including models (and entities to some extent.)
+
+Here's more complex example that accepts data encapsulated in model instances and returns model instances:
+
+```ruby
+#!/usr/bin/env ruby
+
+require "foobara/load_dotenv"
+
+Foobara::LoadDotenv.run!(env: "development", dir: __dir__)
+
+require "foobara/anthropic_api"
+# require "foobara/open_ai_api"
+# require "foobara/ollama_api"
+require "foobara/llm_backed_command"
+
+class PossibleUsState < Foobara::Model
+  attributes do
+    name :string, :required,
+         "A name that potentially might be a name of a US state, spelled correctly or incorrectly"
+  end
+end
+
+class VerifiedUsState < Foobara::Model
+  attributes do
+    possible_us_state PossibleUsState, :required,
+                      "The original possible US state that was passed in"
+    spelling_correction_required :boolean, :required, "Whether or not the original spelling was correct"
+    corrected_spelling :string, :allow_nil,
+                       "If the original spelling was incorrect, the corrected spelling will be here"
+  end
+end
+
+class SelectUsStateNamesAndCorrectTheirSpelling < Foobara::LlmBackedCommand
+  description <<~DESCRIPTION
+    Accepts a list of possible US state names and sorts them into verified to be the name of a
+    US state and rejected to be the name of a non-US state, as well as correcting the spelling of
+    the US state name if it's not correct.
+
+    example:
+
+    If you pass in ["Kalifornia", "Los Angeles", "New York"] the result will be:
+
+    result[:verified].length # => 2
+    result[:verified][0].possible_us_state.name # => "Kalifornia"
+    result[:verified][0].spelling_correction_required # => true#{" "}
+    result[:verified][0].corrected_spelling # => "California"
+    result[:verified][1].possible_us_state.name # => "New York"
+    result[:verified][1].spelling_correction_required # => false
+    result[:verified][1].corrected_spelling # => nil
+
+    result[:rejected].length # => 1
+    result[:rejected][0].name # => "Los Angeles"
+  DESCRIPTION
+
+  inputs do
+    list_of_possible_us_states [PossibleUsState]
+    llm_model :string, default: "claude-3-7-sonnet-20250219"
+  end
+
+  result do
+    verified [VerifiedUsState]
+    rejected [PossibleUsState]
+  end
+end
+
+list_of_possible_us_states = [
+  PossibleUsState.new(name: "Grand Rapids"),
+  PossibleUsState.new(name: "Oregon"),
+  PossibleUsState.new(name: "Yutah"),
+  PossibleUsState.new(name: "Misisipi"),
+  PossibleUsState.new(name: "Tacoma"),
+  PossibleUsState.new(name: "Kalifornia"),
+  PossibleUsState.new(name: "Los Angeles"),
+  PossibleUsState.new(name: "New York")
+]
+
+command = SelectUsStateNamesAndCorrectTheirSpelling.new(list_of_possible_us_states:)
+result = command.run!
+
+puts "Considering:"
+list_of_possible_us_states.each do |possible_us_state|
+  puts "  #{possible_us_state.name}"
+end
+puts
+
+puts "#{result[:verified].length} were verified as US states:"
+result[:verified].each do |verified_us_state|
+  puts "  #{verified_us_state.corrected_spelling || verified_us_state.possible_us_state.name}"
+  if verified_us_state.spelling_correction_required
+    puts "    original incorrect spelling: #{verified_us_state.possible_us_state.name}"
+  end
+end
+
+puts
+puts "#{result[:rejected].length} were rejected as non-US states:"
+result[:rejected].each do |rejected_us_state|
+  puts "  #{rejected_us_state.name}"
+end
+```
+
+This script outputs:
+
+```
+Considering:
+  Grand Rapids
+  Oregon
+  Yutah
+  Misisipi
+  Tacoma
+  Kalifornia
+  Los Angeles
+  New York
+
+5 were verified as US states:
+  Oregon
+  Utah
+    original incorrect spelling: Yutah
+  Mississippi
+    original incorrect spelling: Misisipi
+  California
+    original incorrect spelling: Kalifornia
+  New York
+
+3 were rejected as non-US states:
+  Grand Rapids
+  Tacoma
+  Los Angeles
+```
+
+Note that we were able to access model attributes by methods just fine, and we didn't write any logic on how
+to spell check or any logic on how determine what is or is not a US state.
+
+### Use with entities
+
+Unclear how practical using entities in this way would be, but, here's an interesting
+example of asking a question about some persisted records:
+
+```ruby
+#!/usr/bin/env ruby
+
+require "foobara/load_dotenv"
+
+Foobara::LoadDotenv.run!(env: "development", dir: __dir__)
+
+require "foobara/llm_backed_command"
+require "foobara/sh_cli_connector"
+require "foobara/local_files_crud_driver"
+
+Foobara::Persistence.default_crud_driver = Foobara::LocalFilesCrudDriver.new
+
+class Capybara < Foobara::Entity
+  attributes do
+    id :integer
+    name :string, :required
+    age :integer, :required
+  end
+  primary_key :id
+end
+
+class CreateCapybara < Foobara::Command
+  inputs Capybara.attributes_for_create
+  result Capybara
+
+  def execute
+    create_capybara
+
+    capybara
+  end
+
+  attr_accessor :capybara
+
+  def create_capybara
+    self.capybara = Capybara.create(inputs)
+  end
+end
+
+class SelectOldestCapybara < Foobara::LlmBackedCommand
+  inputs capybaras: [Capybara]
+  result Capybara
+end
+
+connector = Foobara::CommandConnectors::ShCliConnector.new
+
+connector.connect(CreateCapybara)
+connector.connect(SelectOldestCapybara)
+
+connector.run(ARGV)
+```
+
+We can create some capybara records like this:
+
+```
+$ ./oldest-capybara-example CreateCapybara --age 100 --name Fumiko
+age: 100,
+name: "Fumiko",
+id: 1
+$ ./oldest-capybara-example CreateCapybara --age 200 --name Barbara
+age: 200,
+name: "Barbara",
+id: 2
+$ ./oldest-capybara-example CreateCapybara --age 300 --name Basil
+age: 300,
+name: "Basil",
+id: 3
+```
+
+And then ask who is older, Barbara or Basil, like so:
+
+```
+$ ./oldest-capybara-example SelectOldestCapybara --capybaras 2 3
+id: 3,
+name: "Basil",
+age: 300
+```
+
+Basil is older.  Pretty crazy we didn't have to write any logic about age comparisons at all.
+And our SelectOldestCapybara command is only 4 lines long as a result.
+
+### Complete scripts to play with
+
+There are various scripts in [example_scripts/shorter](example_scripts/shorter)
+and [example_scripts/higher_quality](example_scripts/higher_quality) that you can play with.
+
+The difference between the two directories is that `higher_quality/` contains scripts that are more realistic
+with more descriptions and comments and `shorter/` focuses on keeping the scripts short and simple.
+
+Those directories also have a Gemfile each if helpful for pulling in dependencies
+and so that you can use `bundle exec` there if needed.
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub
-at https://github.com/foobara/llm-backed-command
+Bug reports and pull requests are welcome on GitHub at https://github.com/foobara/llm-backed-command
 
 ## License
 

data/lib/foobara/llm_backed_command.rb

@@ -4,6 +4,13 @@ require "foobara/command_connectors"
 require "foobara/ai"
 require "foobara/json_schema_generator"
 
+if Foobara::Ai.foobara_all_command.empty?
+  # :nocov:
+  raise "No api services loaded. " \
+        "Did you forget to set a URL/API key env var or a require for either ollama, anthropic, or openai?"
+  # :nocov:
+end
+
 Foobara::Util.require_directory "#{__dir__}/../../src"
 
 Foobara::Monorepo.project "llm_backed_command", project_path: "#{__dir__}/../../"

data/src/llm_backed_command.rb

@@ -1,176 +1,5 @@
-# NOTE: You can add the following inputs if you'd like, or, create methods with these names
-# on the class.
-#
-# inputs do
-#   association_depth :symbol, one_of: JsonSchemaGenerator::AssociationDepth, default: AssociationDepth::ATOM
-#   llm_model :symbol, one_of: Foobara::Ai::AnswerBot::Types::ModelEnum
-# end
 module Foobara
-  module LlmBackedCommand
-    include Concern
-
-    on_include do
-      depends_on Ai::AnswerBot::Ask
-    end
-
-    def execute
-      determine_serializer
-      construct_input_json
-      generate_answer
-      parse_answer
-
-      parsed_answer
-    end
-
-    attr_accessor :serializer, :input_json, :answer, :parsed_answer
-
-    def determine_serializer
-      depth = if respond_to?(:association_depth)
-                association_depth
-              else
-                Foobara::JsonSchemaGenerator::AssociationDepth::AGGREGATE
-              end
-
-      serializer = case depth
-                   when Foobara::JsonSchemaGenerator::AssociationDepth::ATOM
-                     Foobara::CommandConnectors::Serializers::AtomicSerializer
-                   when Foobara::JsonSchemaGenerator::AssociationDepth::AGGREGATE
-                     Foobara::CommandConnectors::Serializers::AggregateSerializer
-                   when Foobara::JsonSchemaGenerator::AssociationDepth::PRIMARY_KEY_ONLY
-                     # :nocov:
-                     raise "PRIMARY_KEY_ONLY depth not yet implemented"
-                   # :nocov:
-                   else
-                     # :nocov:
-                     raise "Unknown depth: #{depth}"
-                     # :nocov:
-                   end
-
-      # cache this?
-      self.serializer = serializer.new
-    end
-
-    def construct_input_json
-      inputs_without_llm_integration_inputs = inputs.except(:llm_model, :association_depth)
-      input_json = serializer.serialize(inputs_without_llm_integration_inputs)
-
-      self.input_json = JSON.fast_generate(input_json)
-    end
-
-    def generate_answer
-      ask_inputs = {
-        instructions: llm_instructions,
-        question: input_json
-      }
-
-      if respond_to?(:llm_model)
-        ask_inputs[:model] = llm_model
-      end
-
-      self.answer = run_subcommand!(Ai::AnswerBot::Ask, ask_inputs)
-    end
-
-    def llm_instructions
-      self.class.llm_instructions
-    end
-
-    def parse_answer
-      stripped_answer = answer.gsub(/<THINK>.*?<\/THINK>/mi, "")
-      fencepostless_answer = stripped_answer.gsub(/^\s*```\w*\n(.*)```\s*\z/m, "\\1")
-      # TODO: should we verify against json-schema or no?
-      self.parsed_answer = begin
-        JSON.parse(fencepostless_answer)
-      rescue => e
-        # see if we can extract the last fence-posts content just in case
-        last_fence_post_regex = /```\w*\s*\n((?:(?!```).)+)\n```(?:(?!```).)*\z/m
-        begin
-          match = last_fence_post_regex.match(stripped_answer)
-          if match
-            JSON.parse(match[1])
-          else
-            # :nocov:
-            raise e
-            # :nocov:
-          end
-        rescue
-          # :nocov:
-          raise e
-          # :nocov:
-        end
-      end
-    end
-
-    module ClassMethods
-      def inputs_json_schema
-        @inputs_json_schema ||= JsonSchemaGenerator.to_json_schema(inputs_type_without_llm_integration_inputs)
-      end
-
-      def inputs_type_without_llm_integration_inputs
-        return @inputs_type_without_llm_integration_inputs if @inputs_type_without_llm_integration_inputs
-
-        type_declaration = Util.deep_dup(inputs_type.declaration_data)
-
-        element_type_declarations = type_declaration[:element_type_declarations]
-
-        changed = false
-
-        if element_type_declarations.key?(:llm_model)
-          changed = true
-          element_type_declarations.delete(:llm_model)
-        end
-
-        if element_type_declarations.key?(:association_depth)
-          changed = true
-          element_type_declarations.delete(:association_depth)
-        end
-
-        if type_declaration.key?(:defaults)
-          if type_declaration[:defaults].key?(:llm_model)
-            changed = true
-            type_declaration[:defaults].delete(:llm_model)
-          end
-
-          if type_declaration[:defaults].key?(:association_depth)
-            changed = true
-            type_declaration[:defaults].delete(:association_depth)
-          end
-          if type_declaration[:defaults].empty?
-            type_declaration.delete(:defaults)
-          end
-        end
-
-        @inputs_type_without_llm_integration_inputs = if changed
-                                                        domain.foobara_type_from_declaration(type_declaration)
-                                                      else
-                                                        inputs_type
-                                                      end
-      end
-
-      def result_json_schema
-        @result_json_schema ||= JsonSchemaGenerator.to_json_schema(result_type)
-      end
-
-      def llm_instructions
-        @llm_instructions ||= <<~INSTRUCTIONS
-          You are implementing an API for a command named #{scoped_full_name} which has the following description:
-
-          #{description}#{"  "}
-
-          Here is the inputs JSON schema for the data you will receive:
-
-          #{inputs_json_schema}
-
-          Here is the result JSON schema:
-
-          #{result_json_schema}
-
-          You will receive 1 message containing only JSON data according to the inputs JSON schema above
-          and you will generate a JSON response that is a valid response according to the result JSON schema above.
-
-          You will reply with nothing more than the JSON you've generated so that the calling code
-          can successfully parse your answer.
-        INSTRUCTIONS
-      end
-    end
+  class LlmBackedCommand < Foobara::Command
+    include LlmBackedExecuteMethod
   end
 end

data/src/llm_backed_execute_method.rb

@@ -0,0 +1,176 @@
+# NOTE: You can add the following inputs if you'd like, or, create methods with these names
+# on the class.
+#
+# inputs do
+#   association_depth :symbol, one_of: JsonSchemaGenerator::AssociationDepth, default: AssociationDepth::ATOM
+#   llm_model :symbol, one_of: Foobara::Ai::AnswerBot::Types::ModelEnum
+# end
+module Foobara
+  module LlmBackedExecuteMethod
+    include Concern
+
+    on_include do
+      depends_on Ai::AnswerBot::Ask
+    end
+
+    def execute
+      determine_serializer
+      construct_input_json
+      generate_answer
+      parse_answer
+
+      parsed_answer
+    end
+
+    attr_accessor :serializer, :input_json, :answer, :parsed_answer
+
+    def determine_serializer
+      depth = if respond_to?(:association_depth)
+                association_depth
+              else
+                Foobara::JsonSchemaGenerator::AssociationDepth::AGGREGATE
+              end
+
+      serializer = case depth
+                   when Foobara::JsonSchemaGenerator::AssociationDepth::ATOM
+                     Foobara::CommandConnectors::Serializers::AtomicSerializer
+                   when Foobara::JsonSchemaGenerator::AssociationDepth::AGGREGATE
+                     Foobara::CommandConnectors::Serializers::AggregateSerializer
+                   when Foobara::JsonSchemaGenerator::AssociationDepth::PRIMARY_KEY_ONLY
+                     # :nocov:
+                     raise "PRIMARY_KEY_ONLY depth not yet implemented"
+                   # :nocov:
+                   else
+                     # :nocov:
+                     raise "Unknown depth: #{depth}"
+                     # :nocov:
+                   end
+
+      # cache this?
+      self.serializer = serializer.new
+    end
+
+    def construct_input_json
+      inputs_without_llm_integration_inputs = inputs.except(:llm_model, :association_depth)
+      input_json = serializer.serialize(inputs_without_llm_integration_inputs)
+
+      self.input_json = JSON.fast_generate(input_json)
+    end
+
+    def generate_answer
+      ask_inputs = {
+        instructions: llm_instructions,
+        question: input_json
+      }
+
+      if respond_to?(:llm_model)
+        ask_inputs[:model] = llm_model
+      end
+
+      self.answer = run_subcommand!(Ai::AnswerBot::Ask, ask_inputs)
+    end
+
+    def llm_instructions
+      self.class.llm_instructions
+    end
+
+    def parse_answer
+      stripped_answer = answer.gsub(/<THINK>.*?<\/THINK>/mi, "")
+      fencepostless_answer = stripped_answer.gsub(/^\s*```\w*\n(.*)```\s*\z/m, "\\1")
+      # TODO: should we verify against json-schema or no?
+      self.parsed_answer = begin
+        JSON.parse(fencepostless_answer)
+      rescue => e
+        # see if we can extract the last fence-posts content just in case
+        last_fence_post_regex = /```\w*\s*\n((?:(?!```).)+)\n```(?:(?!```).)*\z/m
+        begin
+          match = last_fence_post_regex.match(stripped_answer)
+          if match
+            JSON.parse(match[1])
+          else
+            # :nocov:
+            raise e
+            # :nocov:
+          end
+        rescue
+          # :nocov:
+          raise e
+          # :nocov:
+        end
+      end
+    end
+
+    module ClassMethods
+      def inputs_json_schema
+        @inputs_json_schema ||= JsonSchemaGenerator.to_json_schema(inputs_type_without_llm_integration_inputs)
+      end
+
+      def inputs_type_without_llm_integration_inputs
+        return @inputs_type_without_llm_integration_inputs if @inputs_type_without_llm_integration_inputs
+
+        type_declaration = Util.deep_dup(inputs_type.declaration_data)
+
+        element_type_declarations = type_declaration[:element_type_declarations]
+
+        changed = false
+
+        if element_type_declarations.key?(:llm_model)
+          changed = true
+          element_type_declarations.delete(:llm_model)
+        end
+
+        if element_type_declarations.key?(:association_depth)
+          changed = true
+          element_type_declarations.delete(:association_depth)
+        end
+
+        if type_declaration.key?(:defaults)
+          if type_declaration[:defaults].key?(:llm_model)
+            changed = true
+            type_declaration[:defaults].delete(:llm_model)
+          end
+
+          if type_declaration[:defaults].key?(:association_depth)
+            changed = true
+            type_declaration[:defaults].delete(:association_depth)
+          end
+          if type_declaration[:defaults].empty?
+            type_declaration.delete(:defaults)
+          end
+        end
+
+        @inputs_type_without_llm_integration_inputs = if changed
+                                                        domain.foobara_type_from_declaration(type_declaration)
+                                                      else
+                                                        inputs_type
+                                                      end
+      end
+
+      def result_json_schema
+        @result_json_schema ||= JsonSchemaGenerator.to_json_schema(result_type)
+      end
+
+      def llm_instructions
+        @llm_instructions ||= <<~INSTRUCTIONS
+          You are implementing an API for a command named #{scoped_full_name} which has the following description:
+
+          #{description}#{"  "}
+
+          Here is the inputs JSON schema for the data you will receive:
+
+          #{inputs_json_schema}
+
+          Here is the result JSON schema:
+
+          #{result_json_schema}
+
+          You will receive 1 message containing only JSON data according to the inputs JSON schema above
+          and you will generate a JSON response that is a valid response according to the result JSON schema above.
+
+          You will reply with nothing more than the JSON you've generated so that the calling code
+          can successfully parse your answer.
+        INSTRUCTIONS
+      end
+    end
+  end
+end

metadata

@@ -1,42 +1,56 @@
 --- !ruby/object:Gem::Specification
 name: foobara-llm-backed-command
 version: !ruby/object:Gem::Version
-  version: 0.0.4
+  version: 0.0.6
 platform: ruby
 authors:
 - Miles Georgi
 bindir: bin
 cert_chain: []
-date: 2025-03-06 00:00:00.000000000 Z
+date: 2025-05-21 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: foobara
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.92
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.0.92
 - !ruby/object:Gem::Dependency
   name: foobara-ai
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.0.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.0.1
 - !ruby/object:Gem::Dependency
   name: foobara-json-schema-generator
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.0.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.0.1
 email:
 - azimux@gmail.com
 executables: []
@@ -49,6 +63,7 @@ files:
 - README.md
 - lib/foobara/llm_backed_command.rb
 - src/llm_backed_command.rb
+- src/llm_backed_execute_method.rb
 homepage: https://github.com/foobara/llm-backed-command
 licenses:
 - MPL-2.0
@@ -71,7 +86,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.5
+rubygems_version: 3.6.2
 specification_version: 4
 summary: Provides an easy way to implement a command whose logic is managed by an
   LLM