groq 0.2.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8d1461971dedb839a98ceba16edeec695a3fbc48216295314e6c319e5976f621
-  data.tar.gz: ac0437a0a14d79c9faab3c88054100928970606e90997187c6e908e67a67dc8c
+  metadata.gz: a27c810eca4d98436dc29bad4d48332a0f00440b3da642895d15739e43f9d9b0
+  data.tar.gz: 021b1ca81acc6d07d059b49b9e4a4771997fbcf2f5f8b026872a28a17052b936
 SHA512:
-  metadata.gz: 422b5c160196127928397e568aa15e76dc4f63d1388391bce9cae4ad4d6d0b0fb4063fb52126f04a5b32667179532ea62af96f64acb785ffffed875d4c0646cb
-  data.tar.gz: a537f489dedaa533e9fdb444c6e6d3007dab7164c8306260b0409cd5d2b8bf8802373320d7a79016f04a8e71295845f2504162aad8d7f4997db30c2cad7e32f5
+  metadata.gz: beafd45e9e8d1fa716f7fbe409b700d5759d55969fc35fef905aff1e02a340b969170ebc4c19ae267755df74325aa88ad9076cb799877de6497b9c2bd393b6df
+  data.tar.gz: f99f6a845c58eec585508f7e90bb88717152d72a5e535462e2d2a470fd1742974968dac0c901626d03acc39a4cbef5a3fe43a00c49c1a87d66ad0ea966c5bd3d

data/README.md

@@ -69,7 +69,6 @@ If bundler is not being used to manage dependencies, install the gem by executin
 ```plain
 gem install groq
 ```
-
 ## Usage
 
 - Get your API key from [console.groq.com/keys](https://console.groq.com/keys)
@@ -80,13 +79,19 @@ gem install groq
 client = Groq::Client.new # uses ENV["GROQ_API_KEY"] and "llama3-8b-8192"
 client = Groq::Client.new(api_key: "...", model_id: "llama3-8b-8192")
 
-Groq.configuration do |config|
+Groq.configure do |config|
   config.api_key = "..."
   config.model_id = "llama3-70b-8192"
 end
 client = Groq::Client.new
 ```
 
+In a Rails application, you can generate a `config/initializer/groq.rb` file with:
+
+```plain
+rails g groq:install
+```
+
 There is a simple chat function to send messages to a model:
 
 ```ruby
@@ -166,7 +171,7 @@ As above, you can specify the default model to use for all `chat()` calls:
 ```ruby
 client = Groq::Client.new(model_id: "llama3-70b-8192")
 # or
-Groq.configuration do |config|
+Groq.configure do |config|
   config.model_id = "llama3-70b-8192"
 end
 ```
@@ -190,9 +195,8 @@ end
 The output might looks similar to:
 
 ```plain
-User message: Hello, world!
+> User message: Hello, world!
 Assistant reply with model llama3-8b-8192:
-{"role"=>"assistant", "content"=>"Hello, world! It's great to meet you! Is there something I can help you with, or would you like to chat?"}
 Assistant reply with model llama3-70b-8192:
 {"role"=>"assistant", "content"=>"The classic \"Hello, world!\" It's great to see you here! Is there something I can help you with, or would you like to just chat?"}
 Assistant reply with model llama2-70b-4096:
@@ -227,6 +231,33 @@ JSON.parse(response["content"])
 # => {"number"=>7}
 ```
 
+### Using dry-schema with JSON mode
+
+As a bonus, the `S` or `System` helper can take a `json_schema:` argument and the system message will include the `JSON` keyword and the formatted schema in its content.
+
+For example, if you're using [dry-schema](https://dry-rb.org/gems/dry-schema/1.13/extensions/json_schema/) with its `:json_schema` extension you can use Ruby to describe JSON schema.
+
+```ruby
+require "dry-schema"
+Dry::Schema.load_extensions(:json_schema)
+
+person_schema_defn = Dry::Schema.JSON do
+  required(:name).filled(:string)
+  optional(:age).filled(:integer)
+  optional(:email).filled(:string)
+end
+person_schema = person_schema_defn.json_schema
+
+response = @client.chat([
+  S("You're excellent at extracting personal information", json_schema: person_schema),
+  U("I'm Dr Nic and I'm almost 50.")
+], json: true)
+JSON.parse(response["content"])
+# => {"name"=>"Dr Nic", "age"=>49}
+```
+
+NOTE: `bin/console` already loads the `dry-schema` library and the `json_schema` extension because its handy.
+
 ### Tools/Functions
 
 LLMs are increasingly supporting deferring to tools or functions to fetch data, perform calculations, or store structured data. Groq Cloud in turn then supports their tool implementations through its API.
@@ -298,10 +329,10 @@ The defaults are:
 => 1
 ```
 
-You can override them in the `Groq.configuration` block, or with each `chat()` call:
+You can override them in the `Groq.configure` block, or with each `chat()` call:
 
 ```ruby
-Groq.configuration do |config|
+Groq.configure do |config|
   config.max_tokens = 512
   config.temperature = 0.5
 end
@@ -309,6 +340,273 @@ end
 @client.chat("Hello, world!", max_tokens: 512, temperature: 0.5)
 ```
 
+### Debugging API calls
+
+The underlying HTTP library being used is faraday, and you can enabled debugging, or configure other faraday internals by passing a block to the `Groq::Client.new` constructor.
+
+```ruby
+require 'logger'
+
+# Create a logger instance
+logger = Logger.new(STDOUT)
+logger.level = Logger::DEBUG
+
+@client = Groq::Client.new do |faraday|
+  # Log request and response bodies
+  faraday.response :logger, logger, bodies: true
+end
+```
+
+If you pass `--debug` to `bin/console` you will have this logger setup for you.
+
+```plain
+bin/console --debug
+```
+
+### Streaming
+
+If your AI assistant responses are being telecast live to a human, then that human might want some progressive responses. The Groq API supports streaming responses.
+
+Pass a block to `chat()` with either one or two arguments.
+
+1. The first argument is the string content chunk of the response.
+2. The optional second argument is the full response object from the API containing extra metadata.
+
+The final block call will be the last chunk of the response:
+
+1. The first argument will be `nil`
+2. The optional second argument, the full response object, contains a summary of the Groq API usage, such as prompt tokens, prompt time, etc.
+
+```ruby
+puts "🍕 "
+messages = [
+  S("You are a pizza sales person."),
+  U("What do you sell?")
+]
+@client.chat(messages) do |content|
+  print content
+end
+puts
+```
+
+Each chunk of the response will be printed to the console as it is received. It will look pretty.
+
+The default `llama3-7b-8192` model is very very fast and you might not see any streaming. Try a slower model like `llama3-70b-8192` or `mixtral-8x7b-32768`.
+
+```ruby
+@client = Groq::Client.new(model_id: "llama3-70b-8192")
+@client.chat("Write a long poem about patience") do |content|
+  print content
+end
+puts
+```
+
+You can pass in a second argument to get the full response JSON object:
+
+```ruby
+@client.chat("Write a long poem about patience") do |content, response|
+  pp content
+  pp response
+end
+```
+
+Alternately, you can pass a `Proc` or any object that responds to `call` via a `stream:` keyword argument:
+
+```ruby
+@client.chat("Write a long poem about patience", stream: ->(content) { print content })
+```
+
+You could use a class with a `call` method with either one or two arguments, like the `Proc` discussion above.
+
+```ruby
+class MessageBits
+  def initialize(emoji)
+    print "#{emoji} "
+    @bits = []
+  end
+
+  def call(content)
+    if content.nil?
+      puts
+    else
+      print(content)
+      @bits << content
+    end
+  end
+
+  def to_s
+    @bits.join("")
+  end
+
+  def to_assistant_message
+    Assistant(to_s)
+  end
+end
+
+bits = MessageBits.new("🍕")
+@client.chat("Write a long poem about pizza", stream: bits)
+```
+
+## Examples
+
+Here are some example uses of Groq, of the `groq` gem and its syntax.
+
+Also, see the [`examples/`](examples/) folder for more example apps.
+
+### Pizzeria agent
+
+Talking with a pizzeria.
+
+Our pizzeria agent can be as simple as a function that combines a system message and the current messages array:
+
+```ruby
+@agent_message = <<~EOS
+  You are an employee at a pizza store.
+
+  You sell hawaiian, and pepperoni pizzas; in small and large sizes for $10, and $20 respectively.
+
+  Pick up only in. Ready in 10 mins. Cash on pickup.
+EOS
+
+def chat_pizza_agent(messages)
+  @client.chat([
+    System(@agent_message),
+    *messages
+  ])
+end
+```
+
+Now for our first interaction:
+
+```ruby
+messages = [U("Is this the pizza shop? Do you sell hawaiian?")]
+
+response = chat_pizza_agent(messages)
+puts response["content"]
+```
+
+The output might be:
+
+> Yeah! This is the place! Yes, we sell Hawaiian pizzas here! We've got both small and large sizes available for you. The small Hawaiian pizza is $10, and the large one is $20. Plus, because we're all about getting you your pizza fast, our pick-up time is only 10 minutes! So, what can I get for you today? Would you like to order a small or large Hawaiian pizza?
+
+Continue with user's reply.
+
+Note, we build the `messages` array with the previous user and assistant messages and the new user message:
+
+```ruby
+messages << response << U("Yep, give me a large.")
+response = chat_pizza_agent(messages)
+puts response["content"]
+```
+
+Response:
+
+> I'll get that ready for you. So, to confirm, you'd like to order a large Hawaiian pizza for $20, and I'll have it ready for you in 10 minutes. When you come to pick it up, please have the cash ready as we're a cash-only transaction. See you in 10!
+
+Making a change:
+
+```ruby
+messages << response << U("Actually, make it two smalls.")
+response = chat_pizza_agent(messages)
+puts response["content"]
+```
+
+Response:
+
+> I've got it! Two small Hawaiian pizzas on the way! That'll be $20 for two small pizzas. Same deal, come back in 10 minutes to pick them up, and bring cash for the payment. See you soon!
+
+### Pizza customer agent
+
+Oh my. Let's also have an agent that represents the customer.
+
+```ruby
+@customer_message = <<~EOS
+  You are a customer at a pizza store.
+
+  You want to order a pizza. You can ask about the menu, prices, sizes, and pickup times.
+
+  You'll agree with the price and terms of the pizza order.
+
+  You'll make a choice of the available options.
+
+  If you're first in the conversation, you'll say hello and ask about the menu.
+EOS
+
+def chat_pizza_customer(messages)
+  @client.chat([
+    System(@customer_message),
+    *messages
+  ])
+end
+```
+
+First interaction starts with no user or assistant messages. We're generating the customer's first message:
+
+```ruby
+customer_messages = []
+response = chat_pizza_customer(customer_messages)
+puts response["content"]
+```
+
+Customer's first message:
+
+> Hello! I'd like to order a pizza. Could you tell me more about the menu and prices? What kind of pizzas do you have available?
+
+Now we need to pass this to the pizzeria agent:
+
+```ruby
+customer_message = response["content"]
+pizzeria_messages = [U(customer_message)]
+response = chat_pizza_agent(pizzeria_messages)
+puts response["content"]
+```
+
+Pizzeria agent response:
+
+> Hi there! Yeah, sure thing! We've got two delicious options to choose from: Hawaiian and Pepperoni. Both come in small and large sizes. The small pizzas are $10 and the large pizzas are $20.
+>
+> Our Hawaiian pizza features fresh ham and pineapple on a bed of melted mozzarella. And if you're in the mood for something classic, our Pepperoni pizza is loaded with plenty of sliced pepperoni and melted mozzarella cheese.
+
+Now let's add this response to the customer agent's message array, and generate the customer's next response to the pizzera:
+
+```ruby
+customer_messages << U(response["content"])
+response = chat_pizza_customer(customer_messages)
+puts response["content"]
+```
+
+Customer agent response:
+
+> Wow, those both sound delicious! I'm intrigued by the Hawaiian combo, I never thought of putting ham and pineapple on a pizza before. How would you recommend I customize it? Can I add any extra toppings or keep it as is? And do you have any recommendations for the size? Small or large?
+
+Add this to the pizzeria agent's message array, and generate the pizzeria's response:
+
+```ruby
+pizzeria_messages << U(response["content"])
+response = chat_pizza_agent(pizzeria_messages)
+puts response["content"]
+```
+
+Pizzeria agent response:
+
+> The Hawaiian pizza is definitely a unique twist on traditional toppings! You can definitely customize it to your liking. We allow two extra toppings of your choice for an additional $1 each. If you want to add any other toppings beyond that, it's $2 per topping.
+>
+> As for recommends, I'd say the small size is a great starting point, especially if you're trying something new like the Hawaiian pizza. The small size is $10 and it's a great bite-sized portion. But if you're looking for a bigger pie, the large size is $20 and would be a great option if you're feeding a crowd or want leftovers.
+>
+> Keep in mind that our pizzas are cooked fresh in 10 minutes, so it's ready when it's ready! Would you like to place an order now?
+
+Will the customer actually buy anything now?
+
+> I think I'd like to go with the Hawaiian pizza in the small size, so the total would be $10. And I'll take advantage of the extra topping option. I think I'll add some mushrooms to it. So, that's an extra $1 for the mushroom topping. Would that be $11 total? And do you have a pickup time available soon?
+
+OMG, the customer bought something.
+
+Pizzeria agent response:
+
+> That sounds like a great choice! Yeah, the total would be $11, the small Hawaiian pizza with mushrooms. And yes, we do have pickup available shortly. It'll be ready in about 10 minutes. Cash on pickup, okay? Would you like to pay when you pick up your pizza?
+
+Maybe these two do not know how to stop talking. The Halting Problem exists in pizza shops too.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.

data/examples/README.md

@@ -0,0 +1,120 @@
+# Examples
+
+## User Chat
+
+Chat with a pre-defined agent using the following command:
+
+```bash
+bundle exec examples/user-chat.rb
+# or
+bundle exec examples/user-chat.rb --agent-prompt examples/agent-prompts/helloworld.yml
+```
+
+There are two example agent prompts available:
+
+- `examples/agent-prompts/helloworld.yml` (the default)
+- `examples/agent-prompts/pizzeria-sales.yml`
+
+At the prompt, either talk to the AI agent, or some special commands:
+
+- `exit` to exit the conversation
+- `summary` to get a summary of the conversation so far
+
+### Streaming text chunks
+
+There is also an example of streaming the conversation to terminal as it is received from Groq API.
+
+It defaults to the slower `llama3-70b-8192` model so that the streaming is more noticable.
+
+```bash
+bundle exec examples/user-chat-streaming.rb --agent-prompt examples/agent-prompts/pizzeria-sales.yml
+```
+
+### Streaming useful chunks (e.g. JSON)
+
+If the response is returning a list of objects, such as a sequence of JSON objects, you can try to stream the chunks that make up the JSON objects and process them as soon as they are complete.
+
+```bash
+bundle exec examples/streaming-to-json-objects.rb
+```
+
+This will produce JSON for each planet in the solar system, one at a time. The API does not return each JSON as a chunk, rather it only returns `{` and `"` and `name` as distinct chunks. But the example code [`examples/streaming-to-json-objects.rb`](examples/streaming-to-json-objects.rb) shows how you might build up JSON objects from chunks, and process it (e.g. store to DB) as soon as it is complete.
+
+The system prompt used is:
+
+```plain
+Write out the names of the planets of our solar system, and a brief description of each one.
+
+Return JSON object for each one:
+
+{ "name": "Mercury", "position": 1, "description": "Mercury is ..." }
+
+Between each response, say "NEXT" to clearly delineate each JSON response.
+
+Don't say anything else except the JSON objects above.
+```
+
+The code in the repo uses the `NEXT` token to know when to process the JSON object.
+
+The output will look like, with each JSON object printed (or saved to DB) obly when it has been completely built from chunks.
+
+```json
+{"name":"Mercury","position":1,"description":"Mercury is the smallest planet in our solar system, with a highly elliptical orbit that takes it extremely close to the sun."}
+{"name":"Venus","position":2,"description":"Venus is often called Earth's twin due to their similar size and mass, but it has a thick atmosphere that traps heat, making it the hottest planet."}
+```
+
+### Pizzeria
+
+Run the pizzeria example with the following command:
+
+```bash
+bundle exec examples/user-chat.rb --agent-prompt examples/agent-prompts/pizzeria-sales.yml
+```
+
+> 🍕 Hello! Thank you for calling our pizzeria. I'm happy to help you with your inquiry. Do you have a question about our menu or would you like to place an order?
+>
+> 😋 What's your cheapest?
+>
+> 🍕 Our cheapest menu item is the Garlic Knots (6 pieces), which is served with marinara sauce and priced at $5. Would you like to add it to your order or is there something else that catches your attention?
+>
+> 😋 Yes. And a cheap pizza.
+>
+> 🍕 Our cheapest pizza option is the Veggie Pizza. It comes with tomato sauce, mozzarella, and a mix of bell peppers, onions, and mushrooms, all for $13. Would you like to order the Veggie Pizza and the Garlic Knots together?
+>
+> 😋 Yep
+>
+> 🍕 I've taken note of your order. You're ordering a Veggie Pizza and 6 Garlic Knots. Your total comes out to be $18. Would you like to pay by credit card, cash, or take any other payment method?
+>
+> 😋 summary
+>
+> Here is a summary of the current conversation:
+>
+> The conversation started with a customer calling the pizzeria and speaking with an AI assistant. The assistant offered to help with menu inquiries or taking an order. The customer was considering ordering from the menu. The assistant presented the cheapest menu item, Garlic Knots, and asked if the customer wanted to add it to their order. The customer may have been interested in other options as well. The assistant then presented the cheapest pizza option, the Veggie Pizza, and asked if the customer wanted to order it along with the Garlic Knots. The customer agreed, and the assistant took note of the order, which consisted of a Veggie Pizza and 6 Garlic Knots for a total of $18. The assistant asked how the customer would like to pay for their order.
+>
+> 😋 exit
+
+## Two Agents
+
+Here is an example of two agents talking to each other, without any user input.
+
+### Buying Pizza
+
+```bash
+bundle exec examples/groq-two-agents-chatting.rb --agent-prompt examples/agent-prompts/pizzeria-sales.yml --agent-prompt examples/agent-prompts/food-customer.yml
+```
+
+This will proceed for 10 turns, with the agents taking turns to speak. Pass the `-i 15` option to increase the number of turns.
+
+> 🍕 Hello! Welcome to Pizza Palace. Thank you for reaching out to us. How can I assist you today?
+>
+> 😋 Hi! Thanks for having me. I'm actually pretty hungry, so I'm looking to order some food. Can you tell me a bit about your menu? What are some of your popular items?
+>
+> 🍕 I'd be happy to tell you more about our menu. We have a variety of delicious options to choose from. Our most popular items include our Margherita Pizza, Pepperoni Pizza, and BBQ Chicken Pizza. The Margherita is a classic with tomato sauce, mozzarella, and fresh basil. The Pepperoni Pizza is a crowd-pleaser with a generous layer of pepperoni on top. And our BBQ Chicken Pizza has a sweet and tangy BBQ sauce, topped with chicken, onions, and cilantro.
+>
+> We also have some great non-pizza options, such as our Garlic Knots, which are a favorite among our customers. And for dessert, our Cannoli are a must-try - they're filled with creamy ricotta cheese and chocolate chips.
+>
+> What sounds good to you? Would you like me to walk you through our entire menu or is there something specific you're in the mood for?
+>
+> 😋 Mmm, everything sounds delicious! I think I'll go for something a bit hearty. Can you tell me more about the BBQ Chicken Pizza? What kind of chicken is used? And is the pepperoni on the Pepperoni Pizza thick-cut or thin-cut?
+>
+> Also, how would you recommend ordering the Garlic Knots? Are they a side dish or can I get them as part of a combo?

data/examples/agent-prompts/food-customer.yml

@@ -0,0 +1,12 @@
+---
+name: "Food Customer"
+system_prompt: |-
+  You are a hungry customer looking to order some food.
+
+  You can ask about the menu, place an order, or inquire about delivery options.
+
+  When asked about delivery, you say you'll pick up.
+  When asked about payment, you confirm you'll pay when you pick up.
+  You have $25 to spend.
+agent_emoji: "😋"
+can_go_first: true

data/examples/agent-prompts/helloworld.yml

@@ -0,0 +1,7 @@
+---
+name: "Hello World"
+system_prompt: |-
+  I am a friendly agent who always replies to any prompt
+  with a pleasant "Hello" and wishing them well.
+agent_emoji: "🤖"
+user_emoji: "👤"

data/examples/agent-prompts/pizzeria-sales.yml

@@ -0,0 +1,20 @@
+---
+name: "Pizzeria Sales"
+system_prompt: |-
+  You are a phone operator at a busy pizzeria. Your responsibilities include answering calls and online chats from customers who may ask about the menu, wish to place or change orders, or inquire about opening hours.
+
+  Here are some of our popular menu items:
+
+  <menu>
+  Margherita Pizza: Classic with tomato sauce, mozzarella, and basil - $12
+  Pepperoni Pizza: Tomato sauce, mozzarella, and a generous layer of pepperoni - $14
+  Veggie Pizza: Tomato sauce, mozzarella, and a mix of bell peppers, onions, and mushrooms - $13
+  BBQ Chicken Pizza: BBQ sauce, chicken, onions, and cilantro - $15
+  Garlic Knots (6 pieces): Served with marinara sauce - $5
+  Cannoli: Classic Sicilian dessert filled with sweet ricotta cream - $4 each
+  </menu>
+
+  Your goal is to provide accurate information, confirm order details, and ensure a pleasant customer experience. Please maintain a polite and professional tone, be prompt in your responses, and ensure accuracy in order transmission.
+agent_emoji: "🍕"
+user_emoji: "😋"
+can_go_first: true

data/examples/groq-two-agents-chatting.rb

@@ -0,0 +1,124 @@
+#!/usr/bin/env ruby
+#
+# This is a variation of groq-user-chat.rb but without any user prompting.
+# Just two agents chatting with each other.
+
+require "optparse"
+require "groq"
+require "yaml"
+
+include Groq::Helpers
+
+@options = {
+  model: "llama3-8b-8192",
+  # model: "llama3-70b-8192",
+  agent_prompt_paths: [],
+  timeout: 20,
+  interaction_count: 10 # total count of interactions between agents
+}
+OptionParser.new do |opts|
+  opts.banner = "Usage: ruby script.rb [options]"
+
+  opts.on("-m", "--model MODEL", "Model name") do |v|
+    @options[:model] = v
+  end
+
+  opts.on("-a", "--agent-prompt PATH", "Path to an agent prompt file") do |v|
+    @options[:agent_prompt_paths] << v
+  end
+
+  opts.on("-t", "--timeout TIMEOUT", "Timeout in seconds") do |v|
+    @options[:timeout] = v.to_i
+  end
+
+  opts.on("-d", "--debug", "Enable debug mode") do |v|
+    @options[:debug] = v
+  end
+
+  opts.on("-i", "--interaction-count COUNT", "Total count of interactions between agents") do |v|
+    @options[:interaction_count] = v.to_i
+  end
+end.parse!
+
+raise "New two --agent-prompt paths" if @options[:agent_prompt_paths]&.length&.to_i != 2
+
+def debug?
+  @options[:debug]
+end
+
+# Will be instantiated from the agent prompt file
+class Agent
+  def initialize(args = {})
+    args.each do |k, v|
+      instance_variable_set(:"@#{k}", v)
+    end
+    @messages = [S(@system_prompt)]
+  end
+  attr_reader :messages
+  attr_reader :name, :can_go_first, :user_emoji, :agent_emoji, :system_prompt
+  def can_go_first?
+    @can_go_first
+  end
+
+  def self.load_from_file(path)
+    new(YAML.load_file(path))
+  end
+end
+
+# Read the agent prompt from the file
+agents = @options[:agent_prompt_paths].map do |agent_prompt_path|
+  Agent.load_from_file(agent_prompt_path)
+end
+go_first = agents.find { |agent| agent.can_go_first? } || agents.first
+
+# check that each agent contains a system prompt
+agents.each do |agent|
+  raise "Agent #{agent.name} is missing a system prompt" if agent.system_prompt.nil?
+end
+
+# Initialize the Groq client
+@client = Groq::Client.new(model_id: @options[:model], request_timeout: @options[:timeout]) do |f|
+  if debug?
+    require "logger"
+
+    # Create a logger instance
+    logger = Logger.new($stdout)
+    logger.level = Logger::DEBUG
+
+    f.response :logger, logger, bodies: true  # Log request and response bodies
+  end
+end
+
+puts "Welcome to a conversation between #{agents.map(&:name).join(", ")}. Our first speaker will be #{go_first.name}."
+puts "You can quit by typing 'exit'."
+
+agent_speaking_index = agents.index(go_first)
+loop_count = 0
+
+loop do
+  speaking_agent = agents[agent_speaking_index]
+  # Show speaking agent emoji immediately to indicate request going to Groq API
+  print("#{speaking_agent.agent_emoji} ")
+
+  # Use Groq to generate a response
+  response = @client.chat(speaking_agent.messages)
+
+  # Finish the speaking agent line on screen with message response
+  puts(message = response.dig("content"))
+
+  # speaking agent tracks its own message as the Assistant
+  speaking_agent.messages << A(message)
+
+  # other agent tracks the message as the User
+  other_agents = agents.reject { |agent| agent == speaking_agent }
+  other_agents.each do |agent|
+    agent.messages << U(message)
+  end
+
+  agent_speaking_index = (agent_speaking_index + 1) % agents.length
+  loop_count += 1
+  break if loop_count > @options[:interaction_count]
+rescue Faraday::TooManyRequestsError
+  warn "...\n\nGroq API error: too many requests. Exiting."
+  exit 1
+end

data/examples/streaming-to-json-objects.rb

@@ -0,0 +1,87 @@
+#!/usr/bin/env ruby
+
+require "optparse"
+require "groq"
+require "yaml"
+
+include Groq::Helpers
+
+@options = {
+  model: "llama3-70b-8192",
+  timeout: 20
+}
+OptionParser.new do |opts|
+  opts.banner = "Usage: ruby script.rb [options]"
+
+  opts.on("-m", "--model MODEL", "Model name") do |v|
+    @options[:model] = v
+  end
+
+  opts.on("-t", "--timeout TIMEOUT", "Timeout in seconds") do |v|
+    @options[:timeout] = v.to_i
+  end
+
+  opts.on("-d", "--debug", "Enable debug mode") do |v|
+    @options[:debug] = v
+  end
+end.parse!
+
+raise "Missing --model option" if @options[:model].nil?
+
+# Initialize the Groq client
+@client = Groq::Client.new(model_id: @options[:model], request_timeout: @options[:timeout]) do |f|
+  if @options[:debug]
+    require "logger"
+
+    # Create a logger instance
+    logger = Logger.new($stdout)
+    logger.level = Logger::DEBUG
+
+    f.response :logger, logger, bodies: true  # Log request and response bodies
+  end
+end
+
+prompt = <<~TEXT
+  Write out the names of the planets of our solar system, and a brief description of each one.
+
+  Return JSON object for each one:
+
+  { "name": "Mercury", "position": 1, "description": "Mercury is ..." }
+
+  Between each response, say "NEXT" to clearly delineate each JSON response.
+
+  Don't say anything else except the JSON objects above.
+TEXT
+
+# Handle each JSON object once it has been fully streamed
+
+class PlanetStreamer
+  def initialize
+    @buffer = ""
+  end
+
+  def call(content)
+    if !content || content.include?("NEXT")
+      json = JSON.parse(@buffer)
+
+      # do something with JSON, e.g. save to database
+      puts json.to_json
+
+      # reset buffer
+      @buffer = ""
+      return
+    end
+    # if @buffer is empty; and content is not JSON start {, then ignore + return
+    if @buffer.empty? && !content.start_with?("{")
+      return
+    end
+
+    # build JSON
+    @buffer << content
+  end
+end
+
+streamer = PlanetStreamer.new
+
+@client.chat([S(prompt)], stream: streamer)
+puts

data/examples/user-chat-streaming.rb

@@ -0,0 +1,128 @@
+#!/usr/bin/env ruby
+
+require "optparse"
+require "groq"
+require "yaml"
+
+include Groq::Helpers
+
+@options = {
+  model: "llama3-70b-8192",
+  agent_prompt_path: File.join(File.dirname(__FILE__), "agent-prompts/helloworld.yml"),
+  timeout: 20
+}
+OptionParser.new do |opts|
+  opts.banner = "Usage: ruby script.rb [options]"
+
+  opts.on("-m", "--model MODEL", "Model name") do |v|
+    @options[:model] = v
+  end
+
+  opts.on("-a", "--agent-prompt PATH", "Path to agent prompt file") do |v|
+    @options[:agent_prompt_path] = v
+  end
+
+  opts.on("-t", "--timeout TIMEOUT", "Timeout in seconds") do |v|
+    @options[:timeout] = v.to_i
+  end
+
+  opts.on("-d", "--debug", "Enable debug mode") do |v|
+    @options[:debug] = v
+  end
+end.parse!
+
+raise "Missing --model option" if @options[:model].nil?
+raise "Missing --agent-prompt option" if @options[:agent_prompt_path].nil?
+
+# Read the agent prompt from the file
+agent_prompt = YAML.load_file(@options[:agent_prompt_path])
+user_emoji = agent_prompt["user_emoji"]
+agent_emoji = agent_prompt["agent_emoji"]
+system_prompt = agent_prompt["system_prompt"] || agent_prompt["system"]
+can_go_first = agent_prompt["can_go_first"]
+
+# Initialize the Groq client
+@client = Groq::Client.new(model_id: @options[:model], request_timeout: @options[:timeout]) do |f|
+  if @options[:debug]
+    require "logger"
+
+    # Create a logger instance
+    logger = Logger.new($stdout)
+    logger.level = Logger::DEBUG
+
+    f.response :logger, logger, bodies: true  # Log request and response bodies
+  end
+end
+
+puts "Welcome to the AI assistant! I'll respond to your queries."
+puts "You can quit by typing 'exit'."
+
+def produce_summary(messages)
+  combined = messages.map do |message|
+    if message["role"] == "user"
+      "User: #{message["content"]}"
+    else
+      "Assistant: #{message["content"]}"
+    end
+  end.join("\n")
+  response = @client.chat([
+    S("You are excellent at reading a discourse between a human and an AI assistant and summarising the current conversation."),
+    U("Here is the current conversation:\n\n------\n\n#{combined}")
+  ])
+  puts response["content"]
+end
+
+messages = [S(system_prompt)]
+
+if can_go_first
+  print "#{agent_emoji} "
+  message_bits = []
+  response = @client.chat(messages) do |content|
+    # content == nil on last message; and "" on first message
+    next unless content
+    print(content)
+    message_bits << content
+  end
+  puts
+  messages << A(message_bits.join(""))
+end
+
+class MessageBits
+  def initialize(emoji)
+    print "#{emoji} "
+    @bits = []
+  end
+
+  def call(content)
+    if content.nil?
+      puts
+    else
+      print(content)
+      @bits << content
+    end
+  end
+
+  def to_assistant_message
+    Assistant(@bits.join(""))
+  end
+end
+
+loop do
+  print "#{user_emoji} "
+  user_input = gets.chomp
+
+  break if user_input.downcase == "exit"
+
+  # produce summary
+  if user_input.downcase == "summary"
+    produce_summary(messages)
+    next
+  end
+
+  messages << U(user_input)
+
+  # Use Groq to generate a response
+  message_bits = MessageBits.new(agent_emoji)
+  @client.chat(messages, stream: message_bits)
+  messages << message_bits.to_assistant_message
+end

data/examples/user-chat.rb

@@ -0,0 +1,105 @@
+#!/usr/bin/env ruby
+
+require "optparse"
+require "groq"
+require "yaml"
+
+include Groq::Helpers
+
+@options = {
+  model: "llama3-8b-8192",
+  # model: "llama3-70b-8192",
+  agent_prompt_path: File.join(File.dirname(__FILE__), "agent-prompts/helloworld.yml"),
+  timeout: 20
+}
+OptionParser.new do |opts|
+  opts.banner = "Usage: ruby script.rb [options]"
+
+  opts.on("-m", "--model MODEL", "Model name") do |v|
+    @options[:model] = v
+  end
+
+  opts.on("-a", "--agent-prompt PATH", "Path to agent prompt file") do |v|
+    @options[:agent_prompt_path] = v
+  end
+
+  opts.on("-t", "--timeout TIMEOUT", "Timeout in seconds") do |v|
+    @options[:timeout] = v.to_i
+  end
+
+  opts.on("-d", "--debug", "Enable debug mode") do |v|
+    @options[:debug] = v
+  end
+end.parse!
+
+raise "Missing --model option" if @options[:model].nil?
+raise "Missing --agent-prompt option" if @options[:agent_prompt_path].nil?
+
+# Read the agent prompt from the file
+agent_prompt = YAML.load_file(@options[:agent_prompt_path])
+user_emoji = agent_prompt["user_emoji"]
+agent_emoji = agent_prompt["agent_emoji"]
+system_prompt = agent_prompt["system_prompt"] || agent_prompt["system"]
+can_go_first = agent_prompt["can_go_first"]
+
+# Initialize the Groq client
+@client = Groq::Client.new(model_id: @options[:model], request_timeout: @options[:timeout]) do |f|
+  if @options[:debug]
+    require "logger"
+
+    # Create a logger instance
+    logger = Logger.new($stdout)
+    logger.level = Logger::DEBUG
+
+    f.response :logger, logger, bodies: true  # Log request and response bodies
+  end
+end
+
+puts "Welcome to the AI assistant! I'll respond to your queries."
+puts "You can quit by typing 'exit'."
+
+def produce_summary(messages)
+  combined = messages.map do |message|
+    if message["role"] == "user"
+      "User: #{message["content"]}"
+    else
+      "Assistant: #{message["content"]}"
+    end
+  end.join("\n")
+  response = @client.chat([
+    S("You are excellent at reading a discourse between a human and an AI assistant and summarising the current conversation."),
+    U("Here is the current conversation:\n\n------\n\n#{combined}")
+  ])
+  puts response["content"]
+end
+
+messages = [S(system_prompt)]
+
+if can_go_first
+  response = @client.chat(messages)
+  puts "#{agent_emoji} #{response["content"]}"
+  messages << response
+end
+
+loop do
+  print "#{user_emoji} "
+  user_input = gets.chomp
+
+  break if user_input.downcase == "exit"
+
+  # produce summary
+  if user_input.downcase == "summary"
+    produce_summary(messages)
+    next
+  end
+
+  messages << U(user_input)
+
+  # Use Groq to generate a response
+  response = @client.chat(messages)
+
+  message = response.dig("content")
+  puts "#{agent_emoji} #{message}"
+
+  messages << response
+end

data/lib/generators/groq/install_generator.rb

@@ -0,0 +1,20 @@
+require "rails/generators/base"
+
+module Groq
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+
+      def create_groq_init_file
+        create_file "config/initializers/groq.rb", <<~RUBY
+          # frozen_string_literal: true
+
+          Groq.configure do |config|
+            config.api_key = ENV["GROQ_API_KEY"]
+            config.model_id = "llama3-70b-8192"
+          end
+        RUBY
+      end
+    end
+  end
+end

data/lib/groq/client.rb

@@ -7,6 +7,7 @@ class Groq::Client
     model_id
     max_tokens
     temperature
+    request_timeout
   ].freeze
   attr_reader(*CONFIG_KEYS, :faraday_middleware)
 
@@ -21,8 +22,7 @@ class Groq::Client
     @faraday_middleware = faraday_middleware
   end
 
-  # TODO: support stream: true; or &stream block
-  def chat(messages, model_id: nil, tools: nil, max_tokens: nil, temperature: nil, json: false)
+  def chat(messages, model_id: nil, tools: nil, tool_choice: nil, max_tokens: nil, temperature: nil, json: false, stream: nil, &stream_chunk)
     unless messages.is_a?(Array) || messages.is_a?(String)
       raise ArgumentError, "require messages to be an Array or String"
     end
@@ -33,45 +33,128 @@ class Groq::Client
 
     model_id ||= @model_id
 
+    if stream_chunk ||= stream
+      require "event_stream_parser"
+    end
+
     body = {
       model: model_id,
       messages: messages,
       tools: tools,
+      tool_choice: tool_choice,
       max_tokens: max_tokens || @max_tokens,
       temperature: temperature || @temperature,
-      response_format: json ? {type: "json_object"} : nil
+      response_format: json ? {type: "json_object"} : nil,
+      stream_chunk: stream_chunk
     }.compact
     response = post(path: "/openai/v1/chat/completions", body: body)
-    if response.status == 200
-      response.body.dig("choices", 0, "message")
-    else
-      # TODO: send the response.body back in Error object
-      puts "Error: #{response.status}"
-      pp response.body
-      raise Error, "Request failed with status #{response.status}: #{response.body}"
+    # Configured to raise exceptions on 4xx/5xx responses
+    if response.body.is_a?(Hash)
+      return response.body.dig("choices", 0, "message")
     end
+    response.body
   end
 
   def get(path:)
-    client.get do |req|
-      req.url path
-      req.headers["Authorization"] = "Bearer #{@api_key}"
+    client.get(path) do |req|
+      req.headers = headers
     end
   end
 
   def post(path:, body:)
-    client.post do |req|
-      req.url path
-      req.headers["Authorization"] = "Bearer #{@api_key}"
-      req.body = body
+    client.post(path) do |req|
+      configure_json_post_request(req, body)
     end
   end
 
   def client
-    @client ||= Faraday.new(url: @api_url) do |f|
-      f.request :json # automatically encode the request body as JSON
-      f.response :json # automatically decode JSON responses
-      f.adapter Faraday.default_adapter
+    @client ||= begin
+      connection = Faraday.new(url: @api_url) do |f|
+        f.request :json # automatically encode the request body as JSON
+        f.response :json # automatically decode JSON responses
+        f.response :raise_error # raise exceptions on 4xx/5xx responses
+
+        f.adapter Faraday.default_adapter
+        f.options[:timeout] = request_timeout
+      end
+      @faraday_middleware&.call(connection)
+
+      connection
+    end
+  end
+
+  private
+
+  def headers
+    {
+      "Authorization" => "Bearer #{@api_key}",
+      "User-Agent" => "groq-ruby/#{Groq::VERSION}"
+    }
+  end
+
+  #
+  # Code/ideas borrowed from lib/openai/http.rb in https://github.com/alexrudall/ruby-openai/
+  #
+
+  def configure_json_post_request(req, body)
+    req_body = body.dup
+
+    if body[:stream_chunk].respond_to?(:call)
+      req.options.on_data = to_json_stream(user_proc: body[:stream_chunk])
+      req_body[:stream] = true # Tell Groq to stream
+      req_body.delete(:stream_chunk)
+    elsif body[:stream_chunk]
+      raise ArgumentError, "The stream_chunk parameter must be a Proc or have a #call method"
+    end
+
+    req.headers = headers
+    req.body = req_body
+  end
+
+  # Given a proc, returns an outer proc that can be used to iterate over a JSON stream of chunks.
+  # For each chunk, the inner user_proc is called giving it the JSON object. The JSON object could
+  # be a data object or an error object as described in the OpenAI API documentation.
+  #
+  # @param user_proc [Proc] The inner proc to call for each JSON object in the chunk.
+  # @return [Proc] An outer proc that iterates over a raw stream, converting it to JSON.
+  def to_json_stream(user_proc:)
+    parser = EventStreamParser::Parser.new
+
+    proc do |chunk, _bytes, env|
+      if env && env.status != 200
+        raise_error = Faraday::Response::RaiseError.new
+        raise_error.on_complete(env.merge(body: try_parse_json(chunk)))
+      end
+
+      parser.feed(chunk) do |_type, data|
+        next if data == "[DONE]"
+        chunk = JSON.parse(data)
+        delta = chunk.dig("choices", 0, "delta")
+        content = delta.dig("content")
+        if user_proc.is_a?(Proc)
+          # if user_proc takes one argument, pass the content
+          if user_proc.arity == 1
+            user_proc.call(content)
+          else
+            user_proc.call(content, chunk)
+          end
+        elsif user_proc.respond_to?(:call)
+          # if call method takes one argument, pass the content
+          if user_proc.method(:call).arity == 1
+            user_proc.call(content)
+          else
+            user_proc.call(content, chunk)
+          end
+        else
+          raise ArgumentError, "The stream_chunk parameter must be a Proc or have a #call method"
+        end
+      end
     end
   end
+
+  def try_parse_json(maybe_json)
+    JSON.parse(maybe_json)
+  rescue JSON::ParserError
+    maybe_json
+  end
 end

data/lib/groq/helpers.rb

@@ -13,7 +13,10 @@ module Groq::Helpers
     end
     alias_method :Assistant, :A
 
-    def S(content)
+    def S(content, json_schema: nil)
+      if json_schema
+        content += "\nJSON must use schema: #{json_schema}"
+      end
       {role: "system", content: content}
     end
     alias_method :System, :S

data/lib/groq/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Groq
-  VERSION = "0.2.0"
+  VERSION = "0.3.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: groq
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.1
 platform: ruby
 authors:
 - Dr Nic Williams
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2024-04-20 00:00:00.000000000 Z
+date: 2024-05-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: faraday
@@ -52,6 +52,20 @@ dependencies:
     - - ">"
       - !ruby/object:Gem::Version
         version: '5'
+- !ruby/object:Gem::Dependency
+  name: event_stream_parser
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: vcr
   requirement: !ruby/object:Gem::Requirement
@@ -80,6 +94,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: dry-schema
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.13'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.13'
 description: Client library for Groq API for fast LLM inference.
 email:
 - drnicwilliams@gmail.com
@@ -94,6 +122,15 @@ files:
 - README.md
 - Rakefile
 - docs/images/groq-speed-price-20240421.png
+- examples/README.md
+- examples/agent-prompts/food-customer.yml
+- examples/agent-prompts/helloworld.yml
+- examples/agent-prompts/pizzeria-sales.yml
+- examples/groq-two-agents-chatting.rb
+- examples/streaming-to-json-objects.rb
+- examples/user-chat-streaming.rb
+- examples/user-chat.rb
+- lib/generators/groq/install_generator.rb
 - lib/groq-ruby.rb
 - lib/groq.rb
 - lib/groq/client.rb