langchainrb 0.7.0 → 0.7.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4efc896c1c0fa895ebd11bdb3c4d5604ccd34878aa29472419efca85800072da
-  data.tar.gz: c703f8150c7a6a6cb802260da2eeb95d4f7542cc0c5490794cea844351b4fe7c
+  metadata.gz: 49f95a7d3bf92523a3bb74ffd9c1cff35c258c4ecb9523e75b3be4ffdf333359
+  data.tar.gz: a114fc925963757330e83e9287314b1c363206a31293e788ab8f7cc5f8e82249
 SHA512:
-  metadata.gz: 18b0c48b747978b2a92ecd8d118f31b199d5f58469a5dd55ccbfc3a56a9044faf42784e4760ff9a9fca94da019107629c107a3fe586b7d55243aa92bd1c5b949
-  data.tar.gz: d8513d2018ce48a60fbecc9ca3efb91411385fd71301e0c79ac3612d91b2b504427f2ea19b41991f2f4b43a8bd3b650035684b02a46ae0b114d83343ef5bce18
+  metadata.gz: e0fb4076645a2ba09e0e9012fa2ec84260c5294f59628284baace34ad98b4dc2621c29217890aba7995d21288b68b0eab96a4ad4ba74beb1c41d8e79c296539d
+  data.tar.gz: 2d681b82119d4c4356011bcba6f5590429abdb3bea3049ab4c50ba720320493a64838bc08c6b9b8f16d2b2bd71d445795ae56923074a47b26e9948873460a250

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 ## [Unreleased]
 
+## [0.7.2] - 2023-11-02
+- Azure OpenAI LLM support
+
+## [0.7.1] - 2023-10-26
+- Ragas evals tool to evaluate Retrieval Augmented Generation (RAG) pipelines
+
 ## [0.7.0] - 2023-10-22
 - BREAKING: Moving Rails-specific code to `langchainrb_rails` gem
 

data/README.md

@@ -1,10 +1,10 @@
-💎🔗 LangChain.rb
+💎🔗 Langchain.rb
 ---
 ⚡ Building applications with LLMs through composability ⚡
 
-👨‍💻👩‍💻 CURRENTLY SEEKING PEOPLE TO FORM THE CORE GROUP OF MAINTAINERS WITH
+For deep Rails integration see: [langchainrb_rails](https://github.com/andreibondarev/langchainrb_rails) gem.
 
-:warning: UNDER ACTIVE AND RAPID DEVELOPMENT (MAY BE BUGGY AND UNTESTED)
+Available for paid consulting engagements! [Email me](mailto:andrei@sourcelabs.io).
 
 ![Tests status](https://github.com/andreibondarev/langchainrb/actions/workflows/ci.yml/badge.svg?branch=main)
 [![Gem Version](https://badge.fury.io/rb/langchainrb.svg)](https://badge.fury.io/rb/langchainrb)
@@ -12,9 +12,24 @@
 [![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/andreibondarev/langchainrb/blob/main/LICENSE.txt)
 [![](https://dcbadge.vercel.app/api/server/WDARp7J2n8?compact=true&style=flat)](https://discord.gg/WDARp7J2n8)
 
-
 Langchain.rb is a library that's an abstraction layer on top many emergent AI, ML and other DS tools. The goal is to abstract complexity and difficult concepts to make building AI/ML-supercharged applications approachable for traditional software engineers.
 
+## Explore Langchain.rb
+
+- [Installation](#installation)
+- [Usage](#usage)
+- [Vector Search Databases](#using-vector-search-databases-)
+- [Standalone LLMs](#using-standalone-llms-️)
+- [Prompts](#using-prompts-)
+- [Output Parsers](#using-output-parsers)
+- [Agents](#using-agents-)
+- [Loaders](#loaders-)
+- [Examples](#examples)
+- [Evaluations](#evaluations-evals)
+- [Logging](#logging)
+- [Development](#development)
+- [Discord](#discord)
+
 ## Installation
 
 Install the gem and add to the application's Gemfile by executing:
@@ -182,6 +197,42 @@ qdrant:
 client.llm.functions = functions
 ```
 
+#### Azure
+Add `gem "ruby-openai", "~> 5.2.0"` to your Gemfile.
+
+```ruby
+azure = Langchain::LLM::Azure.new(
+  api_key: ENV["AZURE_API_KEY"],
+  llm_options: {
+    api_type: :azure,
+    api_version: "2023-03-15-preview"
+  },
+  embedding_deployment_url: ENV.fetch("AZURE_EMBEDDING_URI"),
+  chat_deployment_url: ENV.fetch("AZURE_CHAT_URI")
+)
+```
+where `AZURE_EMBEDDING_URI` is e.g. `https://custom-domain.openai.azure.com/openai/deployments/gpt-35-turbo` and `AZURE_CHAT_URI` is e.g. `https://custom-domain.openai.azure.com/openai/deployments/ada-2`
+
+You can pass additional parameters to the constructor, it will be passed to the Azure client:
+```ruby
+azure = Langchain::LLM::Azure.new(
+  api_key: ENV["AZURE_API_KEY"],
+  llm_options: {
+    api_type: :azure,
+    api_version: "2023-03-15-preview",
+    request_timeout: 240 # Optional
+  },
+  embedding_deployment_url: ENV.fetch("AZURE_EMBEDDING_URI"),
+  chat_deployment_url: ENV.fetch("AZURE_CHAT_URI")
+)
+```
+```ruby
+azure.embed(text: "foo bar")
+```
+```ruby
+azure.complete(prompt: "What is the meaning of life?")
+```
+
 #### Cohere
 Add `gem "cohere-ruby", "~> 0.9.6"` to your Gemfile.
 
@@ -333,7 +384,7 @@ prompt = Langchain::Prompt.load_from_path(file_path: "spec/fixtures/prompt/promp
 prompt.input_variables #=> ["adjective", "content"]
 ```
 
-### Using Output Parsers 
+### Using Output Parsers
 
 Parse LLM text responses into structured output, such as JSON.
 
@@ -521,6 +572,32 @@ Langchain::Loader.load('https://www.example.com/file.pdf')
 ## Examples
 Additional examples available: [/examples](https://github.com/andreibondarev/langchainrb/tree/main/examples)
 
+## Evaluations (Evals)
+The Evaluations module is a collection of tools that can be used to evaluate and track the performance of the output products by LLM and your RAG (Retrieval Augmented Generation) pipelines.
+
+### RAGAS
+Ragas helps you evaluate your Retrieval Augmented Generation (RAG) pipelines. The implementation is based on this [paper](https://arxiv.org/abs/2309.15217) and the original Python [repo](https://github.com/explodinggradients/ragas). Ragas tracks the following 3 metrics and assigns the 0.0 - 1.0 scores:
+* Faithfulness - the answer is grounded in the given context.
+* Context Relevance - the retrieved context is focused, containing little to no irrelevant information.
+* Answer Relevance - the generated answer addresses the actual question that was provided.
+
+```ruby
+# We recommend using Langchain::LLM::OpenAI as your llm for Ragas
+ragas = Langchain::Evals::Ragas::Main.new(llm: llm)
+
+# The answer that the LLM generated
+# The question (or the original prompt) that was asked
+# The context that was retrieved (usually from a vectorsearch database)
+ragas.score(answer: "", question: "", context: "")
+# =>
+# {
+#   ragas_score: 0.6601257446503674,
+#   answer_relevance_score: 0.9573145866787608,
+#   context_relevance_score: 0.6666666666666666,
+#   faithfulness_score: 0.5
+# }
+```
+
 ## Logging
 
 LangChain.rb uses standard logging mechanisms and defaults to `:warn` level. Most messages are at info level, but we will add debug or warn statements as needed.

data/lib/langchain/evals/ragas/answer_relevance.rb

@@ -0,0 +1,71 @@
+# freeze_string_literal: true
+
+require "matrix"
+
+module Langchain
+  module Evals
+    module Ragas
+      # Answer Relevance refers to the idea that the generated answer should address the actual question that was provided.
+      # This metric evaluates how closely the generated answer aligns with the initial question or instruction.
+      class AnswerRelevance
+        attr_reader :llm, :batch_size
+
+        # @param llm [Langchain::LLM::*] Langchain::LLM::* object
+        # @param batch_size [Integer] Batch size, i.e., number of generated questions to compare to the original question
+        def initialize(llm:, batch_size: 3)
+          @llm = llm
+          @batch_size = batch_size
+        end
+
+        # @param question [String] Question
+        # @param answer [String] Answer
+        # @return [Float] Answer Relevance score
+        def score(question:, answer:)
+          generated_questions = []
+
+          batch_size.times do |i|
+            prompt = answer_relevance_prompt_template.format(
+              question: question,
+              answer: answer
+            )
+            generated_questions << llm.complete(prompt: prompt).completion
+          end
+
+          scores = generated_questions.map do |generated_question|
+            calculate_similarity(original_question: question, generated_question: generated_question)
+          end
+
+          # Find the mean
+          scores.sum(0.0) / scores.size
+        end
+
+        private
+
+        # @param question_1 [String] Question 1
+        # @param question_2 [String] Question 2
+        # @return [Float] Dot product similarity between the two questions
+        def calculate_similarity(original_question:, generated_question:)
+          original_embedding = generate_embedding(original_question)
+          generated_embedding = generate_embedding(generated_question)
+
+          vector_1 = Vector.elements(original_embedding)
+          vector_2 = Vector.elements(generated_embedding)
+          vector_1.inner_product(vector_2)
+        end
+
+        # @param text [String] Text to generate an embedding for
+        # @return [Array<Float>] Embedding
+        def generate_embedding(text)
+          llm.embed(text: text).embedding
+        end
+
+        # @return [PromptTemplate] PromptTemplate instance
+        def answer_relevance_prompt_template
+          @template ||= Langchain::Prompt.load_from_path(
+            file_path: Langchain.root.join("langchain/evals/ragas/prompts/answer_relevance.yml")
+          )
+        end
+      end
+    end
+  end
+end

data/lib/langchain/evals/ragas/context_relevance.rb

@@ -0,0 +1,46 @@
+# freeze_string_literal: true
+
+require "pragmatic_segmenter"
+
+module Langchain
+  module Evals
+    module Ragas
+      # Context Relevance refers to the idea that the retrieved context should be focused, containing as little irrelevant information as possible.
+      class ContextRelevance
+        attr_reader :llm
+
+        # @param llm [Langchain::LLM::*] Langchain::LLM::* object
+        def initialize(llm:)
+          @llm = llm
+        end
+
+        # @param question [String] Question
+        # @param context [String] Context
+        # @return [Float] Context Relevance score
+        def score(question:, context:)
+          prompt = context_relevance_prompt_template.format(
+            question: question,
+            context: context
+          )
+          sentences = llm.complete(prompt: prompt).completion
+
+          (sentence_count(sentences).to_f / sentence_count(context).to_f)
+        end
+
+        private
+
+        def sentence_count(context)
+          ps = PragmaticSegmenter::Segmenter.new(text: context)
+          ps.segment.length
+        end
+
+        # @return [PromptTemplate] PromptTemplate instance
+        def context_relevance_prompt_template
+          @template ||= Langchain::Prompt.load_from_path(
+            file_path: Langchain.root.join("langchain/evals/ragas/prompts/context_relevance.yml")
+          )
+        end
+      end
+    end
+  end
+end

data/lib/langchain/evals/ragas/critique.rb

@@ -0,0 +1,62 @@
+# freeze_string_literal: true
+
+module Langchain
+  module Evals
+    module Ragas
+      # 123
+      class Critique
+        attr_reader :llm, :criterias
+
+        CRITERIAS = {
+          harmfulness: "Does the submission causes or has the potential to cause harm to individuals, groups, or society at large?",
+          maliciousness: "Is the submission intends to harm, deceive, or exploit users?",
+          coherence: "Does the submission presents ideas, information, or arguments in a logical and organized manner?",
+          correctness: "Is the submission factually accurate and free from errors?",
+          conciseness: "Does the submission conveys information or ideas clearly and efficiently, without unnecessary or redundant details"
+        }
+
+        # @param llm [Langchain::LLM::*] Langchain::LLM::* object
+        # @param criterias [Array<String>] Criterias to evaluate
+        def initialize(llm:, criterias: CRITERIAS.keys)
+          @llm = llm
+          @criterias = criterias
+        end
+
+        # @param question [String] Question
+        # @param answer [String] Answer
+        # @param context [String] Context
+        # @return [Float] Faithfulness score
+        def score(question:, answer:)
+          criterias.each do |criteria|
+            subscore(question: question, answer: answer, criteria: criteria)
+          end
+        end
+
+        private
+
+        def subscore(question:, answer:, criteria:)
+          critique_prompt_template.format(
+            input: question,
+            submission: answer,
+            criteria: criteria
+          )
+        end
+
+        def count_verified_statements(verifications)
+          match = verifications.match(/Final verdict for each statement in order:\s*(.*)/)
+          verdicts = match.captures.first
+          verdicts
+            .split(".")
+            .count { |value| value.strip.to_boolean }
+        end
+
+        # @return [PromptTemplate] PromptTemplate instance
+        def critique_prompt_template
+          @template_one ||= Langchain::Prompt.load_from_path(
+            file_path: Langchain.root.join("langchain/evals/ragas/prompts/critique.yml")
+          )
+        end
+      end
+    end
+  end
+end

data/lib/langchain/evals/ragas/faithfulness.rb

@@ -0,0 +1,83 @@
+# freeze_string_literal: true
+
+module Langchain
+  module Evals
+    module Ragas
+      # Faithfulness refers to the idea that the answer should be grounded in the given context,
+      # ensuring that the retrieved context can act as a justification for the generated answer.
+      # The answer is faithful to the context if the claims that are made in the answer can be inferred from the context.
+      #
+      # Score calculation:
+      # F = |V| / |S|
+      #
+      # F = Faithfulness
+      # |V| = Number of statements that were supported according to the LLM
+      # |S| = Total number of statements extracted.
+      #
+      class Faithfulness
+        attr_reader :llm
+
+        # @param llm [Langchain::LLM::*] Langchain::LLM::* object
+        def initialize(llm:)
+          @llm = llm
+        end
+
+        # @param question [String] Question
+        # @param answer [String] Answer
+        # @param context [String] Context
+        # @return [Float] Faithfulness score
+        def score(question:, answer:, context:)
+          statements = statements_extraction(question: question, answer: answer)
+          statements_count = statements
+            .split("\n")
+            .count
+
+          verifications = statements_verification(statements: statements, context: context)
+          verifications_count = count_verified_statements(verifications)
+
+          (verifications_count.to_f / statements_count.to_f)
+        end
+
+        private
+
+        def count_verified_statements(verifications)
+          match = verifications.match(/Final verdict for each statement in order:\s*(.*)/)
+          verdicts = match.captures.first
+          verdicts
+            .split(".")
+            .count { |value| value.strip.to_boolean }
+        end
+
+        def statements_verification(statements:, context:)
+          prompt = statements_verification_prompt_template.format(
+            statements: statements,
+            context: context
+          )
+          llm.complete(prompt: prompt).completion
+        end
+
+        def statements_extraction(question:, answer:)
+          prompt = statements_extraction_prompt_template.format(
+            question: question,
+            answer: answer
+          )
+          llm.complete(prompt: prompt).completion
+        end
+
+        # @return [PromptTemplate] PromptTemplate instance
+        def statements_verification_prompt_template
+          @template_two ||= Langchain::Prompt.load_from_path(
+            file_path: Langchain.root.join("langchain/evals/ragas/prompts/faithfulness_statements_verification.yml")
+          )
+        end
+
+        # @return [PromptTemplate] PromptTemplate instance
+        def statements_extraction_prompt_template
+          @template_one ||= Langchain::Prompt.load_from_path(
+            file_path: Langchain.root.join("langchain/evals/ragas/prompts/faithfulness_statements_extraction.yml")
+          )
+        end
+      end
+    end
+  end
+end

data/lib/langchain/evals/ragas/main.rb

@@ -0,0 +1,70 @@
+# freeze_string_literal: true
+
+module Langchain
+  module Evals
+    # The RAGAS (Retrieval Augmented Generative Assessment) is a framework for evaluating RAG (Retrieval Augmented Generation) pipelines.
+    # Based on the following research: https://arxiv.org/pdf/2309.15217.pdf
+    module Ragas
+      class Main
+        attr_reader :llm
+
+        def initialize(llm:)
+          @llm = llm
+        end
+
+        # Returns the RAGAS scores, e.g.:
+        # {
+        #   ragas_score: 0.6601257446503674,
+        #   answer_relevance_score: 0.9573145866787608,
+        #   context_relevance_score: 0.6666666666666666,
+        #   faithfulness_score: 0.5
+        # }
+        #
+        # @param question [String] Question
+        # @param answer [String] Answer
+        # @param context [String] Context
+        # @return [Hash] RAGAS scores
+        def score(question:, answer:, context:)
+          answer_relevance_score = answer_relevance.score(question: question, answer: answer)
+          context_relevance_score = context_relevance.score(question: question, context: context)
+          faithfulness_score = faithfulness.score(question: question, answer: answer, context: context)
+
+          {
+            ragas_score: ragas_score(answer_relevance_score, context_relevance_score, faithfulness_score),
+            answer_relevance_score: answer_relevance_score,
+            context_relevance_score: context_relevance_score,
+            faithfulness_score: faithfulness_score
+          }
+        end
+
+        private
+
+        # Overall RAGAS score (harmonic mean): https://github.com/explodinggradients/ragas/blob/1dd363e3e54744e67b0be85962a0258d8121500a/src/ragas/evaluation.py#L140-L143
+        #
+        # @param answer_relevance_score [Float] Answer Relevance score
+        # @param context_relevance_score [Float] Context Relevance score
+        # @param faithfulness_score [Float] Faithfulness score
+        # @return [Float] RAGAS score
+        def ragas_score(answer_relevance_score, context_relevance_score, faithfulness_score)
+          reciprocal_sum = (1.0 / answer_relevance_score) + (1.0 / context_relevance_score) + (1.0 / faithfulness_score)
+          (3 / reciprocal_sum)
+        end
+
+        # @return [Langchain::Evals::Ragas::AnswerRelevance] Class instance
+        def answer_relevance
+          @answer_relevance ||= Langchain::Evals::Ragas::AnswerRelevance.new(llm: llm)
+        end
+
+        # @return [Langchain::Evals::Ragas::ContextRelevance] Class instance
+        def context_relevance
+          @context_relevance ||= Langchain::Evals::Ragas::ContextRelevance.new(llm: llm)
+        end
+
+        # @return [Langchain::Evals::Ragas::Faithfulness] Class instance
+        def faithfulness
+          @faithfulness ||= Langchain::Evals::Ragas::Faithfulness.new(llm: llm)
+        end
+      end
+    end
+  end
+end

data/lib/langchain/evals/ragas/prompts/answer_relevance.yml

@@ -0,0 +1,10 @@
+_type: prompt
+input_variables:
+  - answer
+template: |
+  Generate question for the given answer.
+  Answer: The PSLV-C56 mission is scheduled to be launched on Sunday, 30 July 2023 at 06:30 IST / 01:00 UTC. It will be launched from the Satish Dhawan Space Centre, Sriharikota, Andhra Pradesh, India 
+  Question: When is the scheduled launch date and time for the PSLV-C56 mission, and where will it be launched from?
+
+  Answer: {answer}
+  Question:

data/lib/langchain/evals/ragas/prompts/context_relevance.yml

@@ -0,0 +1,10 @@
+_type: prompt
+input_variables:
+  - question
+  - context
+template: |
+  Please extract relevant sentences from the provided context that is absolutely required answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase "Insufficient Information".  While extracting candidate sentences you're not allowed to make any changes to sentences from given context.
+
+  question:{question}
+  context:\n{context}
+  candidate sentences:\n

data/lib/langchain/evals/ragas/prompts/critique.yml

@@ -0,0 +1,18 @@
+_type: prompt
+input_variables:
+  - input
+  - submission
+  - criteria
+template: |
+  Given a input and submission. Evaluate the submission only using the given criteria. 
+  Think step by step providing reasoning and arrive at a conclusion at the end by generating a Yes or No verdict at the end.
+
+  input: Who was the director of Los Alamos Laboratory?
+  submission: Einstein was the director of  Los Alamos Laboratory.
+  criteria: Is the output written in perfect grammar
+  Here's are my thoughts: the criteria for evaluation is whether the output is written in perfect grammar. In this case, the output is grammatically correct. Therefore, the answer is:\n\nYes
+
+  input: {input}
+  submission: {submission}
+  criteria: {criteria}
+  Here's are my thoughts:

data/lib/langchain/evals/ragas/prompts/faithfulness_statements_extraction.yml

@@ -0,0 +1,9 @@
+_type: prompt
+input_variables:
+  - question
+  - answer
+template: |
+  Given a question and answer, create one or more statements from each sentence in the given answer.
+  question: {question}
+  answer: {answer}
+  statements:\n

data/lib/langchain/evals/ragas/prompts/faithfulness_statements_verification.yml

@@ -0,0 +1,27 @@
+_type: prompt
+input_variables:
+  - statements
+  - context
+template: |
+  Consider the given context and following statements, then determine whether they are supported by the information present in the context.
+  Provide a brief explanation for each statement before arriving at the verdict (Yes/No). Provide a final verdict for each statement in order at the end in the given format.
+  Do not deviate from the specified format.
+
+  Context:\nJohn is a student at XYZ University. He is pursuing a degree in Computer Science. He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. John is a diligent student and spends a significant amount of time studying and completing assignments. He often stays late in the library to work on his projects.
+  statements:\n1. John is majoring in Biology.\n2. John is taking a course on Artificial Intelligence.\n3. John is a dedicated student.\n4. John has a part-time job.\n5. John is interested in computer programming.\n
+  Answer:
+  1. John is majoring in Biology.
+  Explanation: John's major is explicitly mentioned as Computer Science. There is no information suggesting he is majoring in Biology.  Verdict: No.
+  2. John is taking a course on Artificial Intelligence.
+  Explanation: The context mentions the courses John is currently enrolled in, and Artificial Intelligence is not mentioned. Therefore, it cannot be deduced that John is taking a course on AI. Verdict: No.
+  3. John is a dedicated student.
+  Explanation: The prompt states that he spends a significant amount of time studying and completing assignments. Additionally, it mentions that he often stays late in the library to work on his projects, which implies dedication. Verdict: Yes.
+  4. John has a part-time job.
+  Explanation: There is no information given in the context about John having a part-time job. Therefore, it cannot be deduced that John has a part-time job.  Verdict: No.
+  5. John is interested in computer programming.
+  Explanation: The context states that John is pursuing a degree in Computer Science, which implies an interest in computer programming. Verdict: Yes.
+  Final verdict for each statement in order: No. No. Yes. No. Yes.
+
+  context:\n{context}
+  statements:\n{statements}
+  Answer:

data/lib/langchain/llm/azure.rb

@@ -0,0 +1,139 @@
+# frozen_string_literal: true
+
+module Langchain::LLM
+  # LLM interface for Azure OpenAI Service APIs: https://learn.microsoft.com/en-us/azure/ai-services/openai/
+  #
+  # Gem requirements:
+  #    gem "ruby-openai", "~> 5.2.0"
+  #
+  # Usage:
+  #    openai = Langchain::LLM::Azure.new(api_key:, llm_options: {}, embedding_deployment_url: chat_deployment_url:)
+  #
+  class Azure < OpenAI
+    attr_reader :embed_client
+    attr_reader :chat_client
+
+    def initialize(
+      api_key:,
+      llm_options: {},
+      default_options: {},
+      embedding_deployment_url: nil,
+      chat_deployment_url: nil
+    )
+      depends_on "ruby-openai", req: "openai"
+      @embed_client = ::OpenAI::Client.new(
+        access_token: api_key,
+        uri_base: embedding_deployment_url,
+        **llm_options
+      )
+      @chat_client = ::OpenAI::Client.new(
+        access_token: api_key,
+        uri_base: chat_deployment_url,
+        **llm_options
+      )
+      @defaults = DEFAULTS.merge(default_options)
+    end
+
+    #
+    # Generate an embedding for a given text
+    #
+    # @param text [String] The text to generate an embedding for
+    # @param params extra parameters passed to OpenAI::Client#embeddings
+    # @return [Langchain::LLM::OpenAIResponse] Response object
+    #
+    def embed(text:, **params)
+      parameters = {model: @defaults[:embeddings_model_name], input: text}
+
+      validate_max_tokens(text, parameters[:model])
+
+      response = with_api_error_handling do
+        embed_client.embeddings(parameters: parameters.merge(params))
+      end
+
+      Langchain::LLM::OpenAIResponse.new(response)
+    end
+
+    #
+    # Generate a completion for a given prompt
+    #
+    # @param prompt [String] The prompt to generate a completion for
+    # @param params  extra parameters passed to OpenAI::Client#complete
+    # @return [Langchain::LLM::Response::OpenaAI] Response object
+    #
+    def complete(prompt:, **params)
+      parameters = compose_parameters @defaults[:completion_model_name], params
+
+      parameters[:messages] = compose_chat_messages(prompt: prompt)
+      parameters[:max_tokens] = validate_max_tokens(parameters[:messages], parameters[:model])
+
+      response = with_api_error_handling do
+        chat_client.chat(parameters: parameters)
+      end
+
+      Langchain::LLM::OpenAIResponse.new(response)
+    end
+
+    #
+    # Generate a chat completion for a given prompt or messages.
+    #
+    # == Examples
+    #
+    #     # simplest case, just give a prompt
+    #     openai.chat prompt: "When was Ruby first released?"
+    #
+    #     # prompt plus some context about how to respond
+    #     openai.chat context: "You are RubyGPT, a helpful chat bot for helping people learn Ruby", prompt: "Does Ruby have a REPL like IPython?"
+    #
+    #     # full control over messages that get sent, equivilent to the above
+    #     openai.chat messages: [
+    #       {
+    #         role: "system",
+    #         content: "You are RubyGPT, a helpful chat bot for helping people learn Ruby", prompt: "Does Ruby have a REPL like IPython?"
+    #       },
+    #       {
+    #         role: "user",
+    #         content: "When was Ruby first released?"
+    #       }
+    #     ]
+    #
+    #     # few-short prompting with examples
+    #     openai.chat prompt: "When was factory_bot released?",
+    #       examples: [
+    #         {
+    #           role: "user",
+    #           content: "When was Ruby on Rails released?"
+    #         }
+    #         {
+    #           role: "assistant",
+    #           content: "2004"
+    #         },
+    #       ]
+    #
+    # @param prompt [String] The prompt to generate a chat completion for
+    # @param messages [Array<Hash>] The messages that have been sent in the conversation
+    # @param context [String] An initial context to provide as a system message, ie "You are RubyGPT, a helpful chat bot for helping people learn Ruby"
+    # @param examples [Array<Hash>] Examples of messages to provide to the model. Useful for Few-Shot Prompting
+    # @param options [Hash] extra parameters passed to OpenAI::Client#chat
+    # @yield [Hash] Stream responses back one token at a time
+    # @return [Langchain::LLM::OpenAIResponse] Response object
+    #
+    def chat(prompt: "", messages: [], context: "", examples: [], **options, &block)
+      raise ArgumentError.new(":prompt or :messages argument is expected") if prompt.empty? && messages.empty?
+
+      parameters = compose_parameters @defaults[:chat_completion_model_name], options, &block
+      parameters[:messages] = compose_chat_messages(prompt: prompt, messages: messages, context: context, examples: examples)
+
+      if functions
+        parameters[:functions] = functions
+      else
+        parameters[:max_tokens] = validate_max_tokens(parameters[:messages], parameters[:model])
+      end
+
+      response = with_api_error_handling { chat_client.chat(parameters: parameters) }
+
+      return if block
+
+      Langchain::LLM::OpenAIResponse.new(response)
+    end
+  end
+end

data/lib/langchain/llm/base.rb

@@ -8,6 +8,7 @@ module Langchain::LLM
   # Langchain.rb provides a common interface to interact with all supported LLMs:
   #
   # - {Langchain::LLM::AI21}
+  # - {Langchain::LLM::Azure}
   # - {Langchain::LLM::Cohere}
   # - {Langchain::LLM::GooglePalm}
   # - {Langchain::LLM::HuggingFace}

data/lib/langchain/llm/cohere.rb

@@ -19,10 +19,10 @@ module Langchain::LLM
       truncate: "START"
     }.freeze
 
-    def initialize(api_key:, default_options: {})
+    def initialize(api_key, default_options = {})
       depends_on "cohere-ruby", req: "cohere"
 
-      @client = ::Cohere::Client.new(api_key: api_key)
+      @client = ::Cohere::Client.new(api_key)
       @defaults = DEFAULTS.merge(default_options)
     end
 

data/lib/langchain/loader_chunkers/html.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module Langchain
+  module LoaderChunkers
+    class HTML < Base
+      EXTENSIONS = [".html", ".htm"]
+      CONTENT_TYPES = ["text/html"]
+
+      # We only look for headings and paragraphs
+      TEXT_CONTENT_TAGS = %w[h1 h2 h3 h4 h5 h6 p]
+
+      def initialize(*)
+        depends_on "nokogiri"
+      end
+
+      # Parse the document and return the text
+      # @param [File] data
+      # @return [String]
+      def parse(data)
+        Nokogiri::HTML(data.read)
+          .css(TEXT_CONTENT_TAGS.join(","))
+          .map(&:inner_text)
+          .join("\n\n")
+      end
+    end
+  end
+end

data/lib/langchain/utils/cosine_similarity.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module Langchain
+  module Utils
+    class CosineSimilarity
+      attr_reader :vector_a, :vector_b
+
+      # @param vector_a [Array<Float>] First vector
+      # @param vector_b [Array<Float>] Second vector
+      def initialize(vector_a, vector_b)
+        @vector_a = vector_a
+        @vector_b = vector_b
+      end
+
+      # Calculate the cosine similarity between two vectors
+      # @return [Float] The cosine similarity between the two vectors
+      def calculate_similarity
+        return nil unless vector_a.is_a? Array
+        return nil unless vector_b.is_a? Array
+        return nil if vector_a.size != vector_b.size
+
+        dot_product = 0
+        vector_a.zip(vector_b).each do |v1i, v2i|
+          dot_product += v1i * v2i
+        end
+
+        a = vector_a.map { |n| n**2 }.reduce(:+)
+        b = vector_b.map { |n| n**2 }.reduce(:+)
+
+        dot_product / (Math.sqrt(a) * Math.sqrt(b))
+      end
+    end
+  end
+end

data/lib/langchain/vectorsearch/base.rb

@@ -25,8 +25,7 @@ module Langchain::Vectorsearch
   #       url:         ENV["WEAVIATE_URL"],
   #       api_key:     ENV["WEAVIATE_API_KEY"],
   #       index_name:  "Documents",
-  #       llm:         :openai,              # or :cohere, :hugging_face, :google_palm, or :replicate
-  #       llm_api_key: ENV["OPENAI_API_KEY"] # API key for the selected LLM
+  #       llm:         Langchain::LLM::OpenAI.new(api_key:)
   #     )
   #
   #     # You can instantiate other supported vector databases the same way:

data/lib/langchain/vectorsearch/chroma.rb

@@ -9,7 +9,7 @@ module Langchain::Vectorsearch
     #     gem "chroma-db", "~> 0.6.0"
     #
     # Usage:
-    # chroma = Langchain::Vectorsearch::Chroma.new(url:, index_name:, llm:, llm_api_key:, api_key: nil)
+    # chroma = Langchain::Vectorsearch::Chroma.new(url:, index_name:, llm:, api_key: nil)
     #
 
     # Initialize the Chroma client

data/lib/langchain/vectorsearch/pinecone.rb

@@ -9,7 +9,7 @@ module Langchain::Vectorsearch
     #     gem "pinecone", "~> 0.1.6"
     #
     # Usage:
-    #     pinecone = Langchain::Vectorsearch::Pinecone.new(environment:, api_key:, index_name:, llm:, llm_api_key:)
+    #     pinecone = Langchain::Vectorsearch::Pinecone.new(environment:, api_key:, index_name:, llm:)
     #
 
     # Initialize the Pinecone client

data/lib/langchain/vectorsearch/qdrant.rb

@@ -9,7 +9,7 @@ module Langchain::Vectorsearch
     #     gem "qdrant-ruby", "~> 0.9.3"
     #
     # Usage:
-    #     qdrant = Langchain::Vectorsearch::Qdrant.new(url:, api_key:, index_name:, llm:, llm_api_key:)
+    #     qdrant = Langchain::Vectorsearch::Qdrant.new(url:, api_key:, index_name:, llm:)
     #
 
     # Initialize the Qdrant client

data/lib/langchain/vectorsearch/weaviate.rb

@@ -9,7 +9,7 @@ module Langchain::Vectorsearch
     #     gem "weaviate-ruby", "~> 0.8.3"
     #
     # Usage:
-    #     weaviate = Langchain::Vectorsearch::Weaviate.new(url:, api_key:, index_name:, llm:, llm_api_key:)
+    #     weaviate = Langchain::Vectorsearch::Weaviate.new(url:, api_key:, index_name:, llm:)
     #
 
     # Initialize the Weaviate adapter

data/lib/langchain/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Langchain
-  VERSION = "0.7.0"
+  VERSION = "0.7.2"
 end

data/lib/langchain.rb

@@ -3,6 +3,7 @@
 require "logger"
 require "pathname"
 require "colorize"
+require "to_bool"
 require "zeitwerk"
 loader = Zeitwerk::Loader.for_gem
 loader.ignore("#{__dir__}/langchainrb.rb")

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: langchainrb
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.7.2
 platform: ruby
 authors:
 - Andrei Bondarev
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-10-23 00:00:00.000000000 Z
+date: 2023-11-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: baran
@@ -94,6 +94,34 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 0.3.0
+- !ruby/object:Gem::Dependency
+  name: to_bool
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+- !ruby/object:Gem::Dependency
+  name: matrix
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: dotenv-rails
   requirement: !ruby/object:Gem::Requirement
@@ -198,14 +226,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.9.6
+        version: 0.9.7
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.9.6
+        version: 0.9.7
 - !ruby/object:Gem::Dependency
   name: docx
   requirement: !ruby/object:Gem::Requirement
@@ -464,14 +492,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.1.0
+        version: 5.2.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 4.1.0
+        version: 5.2.0
 - !ruby/object:Gem::Dependency
   name: safe_ruby
   requirement: !ruby/object:Gem::Requirement
@@ -561,8 +589,19 @@ files:
 - lib/langchain/conversation/response.rb
 - lib/langchain/data.rb
 - lib/langchain/dependency_helper.rb
+- lib/langchain/evals/ragas/answer_relevance.rb
+- lib/langchain/evals/ragas/context_relevance.rb
+- lib/langchain/evals/ragas/critique.rb
+- lib/langchain/evals/ragas/faithfulness.rb
+- lib/langchain/evals/ragas/main.rb
+- lib/langchain/evals/ragas/prompts/answer_relevance.yml
+- lib/langchain/evals/ragas/prompts/context_relevance.yml
+- lib/langchain/evals/ragas/prompts/critique.yml
+- lib/langchain/evals/ragas/prompts/faithfulness_statements_extraction.yml
+- lib/langchain/evals/ragas/prompts/faithfulness_statements_verification.yml
 - lib/langchain/llm/ai21.rb
 - lib/langchain/llm/anthropic.rb
+- lib/langchain/llm/azure.rb
 - lib/langchain/llm/base.rb
 - lib/langchain/llm/cohere.rb
 - lib/langchain/llm/google_palm.rb
@@ -582,6 +621,7 @@ files:
 - lib/langchain/llm/response/openai_response.rb
 - lib/langchain/llm/response/replicate_response.rb
 - lib/langchain/loader.rb
+- lib/langchain/loader_chunkers/html.rb
 - lib/langchain/output_parsers/base.rb
 - lib/langchain/output_parsers/output_fixing_parser.rb
 - lib/langchain/output_parsers/prompts/naive_fix_prompt.yaml
@@ -607,6 +647,7 @@ files:
 - lib/langchain/tool/ruby_code_interpreter.rb
 - lib/langchain/tool/weather.rb
 - lib/langchain/tool/wikipedia.rb
+- lib/langchain/utils/cosine_similarity.rb
 - lib/langchain/utils/token_length/ai21_validator.rb
 - lib/langchain/utils/token_length/base_validator.rb
 - lib/langchain/utils/token_length/cohere_validator.rb