langchainrb 0.7.0 → 0.7.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -0
- data/README.md +82 -5
- data/lib/langchain/evals/ragas/answer_relevance.rb +71 -0
- data/lib/langchain/evals/ragas/context_relevance.rb +46 -0
- data/lib/langchain/evals/ragas/critique.rb +62 -0
- data/lib/langchain/evals/ragas/faithfulness.rb +83 -0
- data/lib/langchain/evals/ragas/main.rb +70 -0
- data/lib/langchain/evals/ragas/prompts/answer_relevance.yml +10 -0
- data/lib/langchain/evals/ragas/prompts/context_relevance.yml +10 -0
- data/lib/langchain/evals/ragas/prompts/critique.yml +18 -0
- data/lib/langchain/evals/ragas/prompts/faithfulness_statements_extraction.yml +9 -0
- data/lib/langchain/evals/ragas/prompts/faithfulness_statements_verification.yml +27 -0
- data/lib/langchain/llm/azure.rb +139 -0
- data/lib/langchain/llm/base.rb +1 -0
- data/lib/langchain/llm/cohere.rb +2 -2
- data/lib/langchain/loader_chunkers/html.rb +27 -0
- data/lib/langchain/utils/cosine_similarity.rb +34 -0
- data/lib/langchain/vectorsearch/base.rb +1 -2
- data/lib/langchain/vectorsearch/chroma.rb +1 -1
- data/lib/langchain/vectorsearch/pinecone.rb +1 -1
- data/lib/langchain/vectorsearch/qdrant.rb +1 -1
- data/lib/langchain/vectorsearch/weaviate.rb +1 -1
- data/lib/langchain/version.rb +1 -1
- data/lib/langchain.rb +1 -0
- metadata +47 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 49f95a7d3bf92523a3bb74ffd9c1cff35c258c4ecb9523e75b3be4ffdf333359
|
4
|
+
data.tar.gz: a114fc925963757330e83e9287314b1c363206a31293e788ab8f7cc5f8e82249
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e0fb4076645a2ba09e0e9012fa2ec84260c5294f59628284baace34ad98b4dc2621c29217890aba7995d21288b68b0eab96a4ad4ba74beb1c41d8e79c296539d
|
7
|
+
data.tar.gz: 2d681b82119d4c4356011bcba6f5590429abdb3bea3049ab4c50ba720320493a64838bc08c6b9b8f16d2b2bd71d445795ae56923074a47b26e9948873460a250
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,11 @@
|
|
1
1
|
## [Unreleased]
|
2
2
|
|
3
|
+
## [0.7.2] - 2023-11-02
|
4
|
+
- Azure OpenAI LLM support
|
5
|
+
|
6
|
+
## [0.7.1] - 2023-10-26
|
7
|
+
- Ragas evals tool to evaluate Retrieval Augmented Generation (RAG) pipelines
|
8
|
+
|
3
9
|
## [0.7.0] - 2023-10-22
|
4
10
|
- BREAKING: Moving Rails-specific code to `langchainrb_rails` gem
|
5
11
|
|
data/README.md
CHANGED
@@ -1,10 +1,10 @@
|
|
1
|
-
💎🔗
|
1
|
+
💎🔗 Langchain.rb
|
2
2
|
---
|
3
3
|
⚡ Building applications with LLMs through composability ⚡
|
4
4
|
|
5
|
-
|
5
|
+
For deep Rails integration see: [langchainrb_rails](https://github.com/andreibondarev/langchainrb_rails) gem.
|
6
6
|
|
7
|
-
|
7
|
+
Available for paid consulting engagements! [Email me](mailto:andrei@sourcelabs.io).
|
8
8
|
|
9
9
|
![Tests status](https://github.com/andreibondarev/langchainrb/actions/workflows/ci.yml/badge.svg?branch=main)
|
10
10
|
[![Gem Version](https://badge.fury.io/rb/langchainrb.svg)](https://badge.fury.io/rb/langchainrb)
|
@@ -12,9 +12,24 @@
|
|
12
12
|
[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/andreibondarev/langchainrb/blob/main/LICENSE.txt)
|
13
13
|
[![](https://dcbadge.vercel.app/api/server/WDARp7J2n8?compact=true&style=flat)](https://discord.gg/WDARp7J2n8)
|
14
14
|
|
15
|
-
|
16
15
|
Langchain.rb is a library that's an abstraction layer on top many emergent AI, ML and other DS tools. The goal is to abstract complexity and difficult concepts to make building AI/ML-supercharged applications approachable for traditional software engineers.
|
17
16
|
|
17
|
+
## Explore Langchain.rb
|
18
|
+
|
19
|
+
- [Installation](#installation)
|
20
|
+
- [Usage](#usage)
|
21
|
+
- [Vector Search Databases](#using-vector-search-databases-)
|
22
|
+
- [Standalone LLMs](#using-standalone-llms-️)
|
23
|
+
- [Prompts](#using-prompts-)
|
24
|
+
- [Output Parsers](#using-output-parsers)
|
25
|
+
- [Agents](#using-agents-)
|
26
|
+
- [Loaders](#loaders-)
|
27
|
+
- [Examples](#examples)
|
28
|
+
- [Evaluations](#evaluations-evals)
|
29
|
+
- [Logging](#logging)
|
30
|
+
- [Development](#development)
|
31
|
+
- [Discord](#discord)
|
32
|
+
|
18
33
|
## Installation
|
19
34
|
|
20
35
|
Install the gem and add to the application's Gemfile by executing:
|
@@ -182,6 +197,42 @@ qdrant:
|
|
182
197
|
client.llm.functions = functions
|
183
198
|
```
|
184
199
|
|
200
|
+
#### Azure
|
201
|
+
Add `gem "ruby-openai", "~> 5.2.0"` to your Gemfile.
|
202
|
+
|
203
|
+
```ruby
|
204
|
+
azure = Langchain::LLM::Azure.new(
|
205
|
+
api_key: ENV["AZURE_API_KEY"],
|
206
|
+
llm_options: {
|
207
|
+
api_type: :azure,
|
208
|
+
api_version: "2023-03-15-preview"
|
209
|
+
},
|
210
|
+
embedding_deployment_url: ENV.fetch("AZURE_EMBEDDING_URI"),
|
211
|
+
chat_deployment_url: ENV.fetch("AZURE_CHAT_URI")
|
212
|
+
)
|
213
|
+
```
|
214
|
+
where `AZURE_EMBEDDING_URI` is e.g. `https://custom-domain.openai.azure.com/openai/deployments/gpt-35-turbo` and `AZURE_CHAT_URI` is e.g. `https://custom-domain.openai.azure.com/openai/deployments/ada-2`
|
215
|
+
|
216
|
+
You can pass additional parameters to the constructor, it will be passed to the Azure client:
|
217
|
+
```ruby
|
218
|
+
azure = Langchain::LLM::Azure.new(
|
219
|
+
api_key: ENV["AZURE_API_KEY"],
|
220
|
+
llm_options: {
|
221
|
+
api_type: :azure,
|
222
|
+
api_version: "2023-03-15-preview",
|
223
|
+
request_timeout: 240 # Optional
|
224
|
+
},
|
225
|
+
embedding_deployment_url: ENV.fetch("AZURE_EMBEDDING_URI"),
|
226
|
+
chat_deployment_url: ENV.fetch("AZURE_CHAT_URI")
|
227
|
+
)
|
228
|
+
```
|
229
|
+
```ruby
|
230
|
+
azure.embed(text: "foo bar")
|
231
|
+
```
|
232
|
+
```ruby
|
233
|
+
azure.complete(prompt: "What is the meaning of life?")
|
234
|
+
```
|
235
|
+
|
185
236
|
#### Cohere
|
186
237
|
Add `gem "cohere-ruby", "~> 0.9.6"` to your Gemfile.
|
187
238
|
|
@@ -333,7 +384,7 @@ prompt = Langchain::Prompt.load_from_path(file_path: "spec/fixtures/prompt/promp
|
|
333
384
|
prompt.input_variables #=> ["adjective", "content"]
|
334
385
|
```
|
335
386
|
|
336
|
-
### Using Output Parsers
|
387
|
+
### Using Output Parsers
|
337
388
|
|
338
389
|
Parse LLM text responses into structured output, such as JSON.
|
339
390
|
|
@@ -521,6 +572,32 @@ Langchain::Loader.load('https://www.example.com/file.pdf')
|
|
521
572
|
## Examples
|
522
573
|
Additional examples available: [/examples](https://github.com/andreibondarev/langchainrb/tree/main/examples)
|
523
574
|
|
575
|
+
## Evaluations (Evals)
|
576
|
+
The Evaluations module is a collection of tools that can be used to evaluate and track the performance of the output products by LLM and your RAG (Retrieval Augmented Generation) pipelines.
|
577
|
+
|
578
|
+
### RAGAS
|
579
|
+
Ragas helps you evaluate your Retrieval Augmented Generation (RAG) pipelines. The implementation is based on this [paper](https://arxiv.org/abs/2309.15217) and the original Python [repo](https://github.com/explodinggradients/ragas). Ragas tracks the following 3 metrics and assigns the 0.0 - 1.0 scores:
|
580
|
+
* Faithfulness - the answer is grounded in the given context.
|
581
|
+
* Context Relevance - the retrieved context is focused, containing little to no irrelevant information.
|
582
|
+
* Answer Relevance - the generated answer addresses the actual question that was provided.
|
583
|
+
|
584
|
+
```ruby
|
585
|
+
# We recommend using Langchain::LLM::OpenAI as your llm for Ragas
|
586
|
+
ragas = Langchain::Evals::Ragas::Main.new(llm: llm)
|
587
|
+
|
588
|
+
# The answer that the LLM generated
|
589
|
+
# The question (or the original prompt) that was asked
|
590
|
+
# The context that was retrieved (usually from a vectorsearch database)
|
591
|
+
ragas.score(answer: "", question: "", context: "")
|
592
|
+
# =>
|
593
|
+
# {
|
594
|
+
# ragas_score: 0.6601257446503674,
|
595
|
+
# answer_relevance_score: 0.9573145866787608,
|
596
|
+
# context_relevance_score: 0.6666666666666666,
|
597
|
+
# faithfulness_score: 0.5
|
598
|
+
# }
|
599
|
+
```
|
600
|
+
|
524
601
|
## Logging
|
525
602
|
|
526
603
|
LangChain.rb uses standard logging mechanisms and defaults to `:warn` level. Most messages are at info level, but we will add debug or warn statements as needed.
|
@@ -0,0 +1,71 @@
|
|
1
|
+
# freeze_string_literal: true
|
2
|
+
|
3
|
+
require "matrix"
|
4
|
+
|
5
|
+
module Langchain
|
6
|
+
module Evals
|
7
|
+
module Ragas
|
8
|
+
# Answer Relevance refers to the idea that the generated answer should address the actual question that was provided.
|
9
|
+
# This metric evaluates how closely the generated answer aligns with the initial question or instruction.
|
10
|
+
class AnswerRelevance
|
11
|
+
attr_reader :llm, :batch_size
|
12
|
+
|
13
|
+
# @param llm [Langchain::LLM::*] Langchain::LLM::* object
|
14
|
+
# @param batch_size [Integer] Batch size, i.e., number of generated questions to compare to the original question
|
15
|
+
def initialize(llm:, batch_size: 3)
|
16
|
+
@llm = llm
|
17
|
+
@batch_size = batch_size
|
18
|
+
end
|
19
|
+
|
20
|
+
# @param question [String] Question
|
21
|
+
# @param answer [String] Answer
|
22
|
+
# @return [Float] Answer Relevance score
|
23
|
+
def score(question:, answer:)
|
24
|
+
generated_questions = []
|
25
|
+
|
26
|
+
batch_size.times do |i|
|
27
|
+
prompt = answer_relevance_prompt_template.format(
|
28
|
+
question: question,
|
29
|
+
answer: answer
|
30
|
+
)
|
31
|
+
generated_questions << llm.complete(prompt: prompt).completion
|
32
|
+
end
|
33
|
+
|
34
|
+
scores = generated_questions.map do |generated_question|
|
35
|
+
calculate_similarity(original_question: question, generated_question: generated_question)
|
36
|
+
end
|
37
|
+
|
38
|
+
# Find the mean
|
39
|
+
scores.sum(0.0) / scores.size
|
40
|
+
end
|
41
|
+
|
42
|
+
private
|
43
|
+
|
44
|
+
# @param question_1 [String] Question 1
|
45
|
+
# @param question_2 [String] Question 2
|
46
|
+
# @return [Float] Dot product similarity between the two questions
|
47
|
+
def calculate_similarity(original_question:, generated_question:)
|
48
|
+
original_embedding = generate_embedding(original_question)
|
49
|
+
generated_embedding = generate_embedding(generated_question)
|
50
|
+
|
51
|
+
vector_1 = Vector.elements(original_embedding)
|
52
|
+
vector_2 = Vector.elements(generated_embedding)
|
53
|
+
vector_1.inner_product(vector_2)
|
54
|
+
end
|
55
|
+
|
56
|
+
# @param text [String] Text to generate an embedding for
|
57
|
+
# @return [Array<Float>] Embedding
|
58
|
+
def generate_embedding(text)
|
59
|
+
llm.embed(text: text).embedding
|
60
|
+
end
|
61
|
+
|
62
|
+
# @return [PromptTemplate] PromptTemplate instance
|
63
|
+
def answer_relevance_prompt_template
|
64
|
+
@template ||= Langchain::Prompt.load_from_path(
|
65
|
+
file_path: Langchain.root.join("langchain/evals/ragas/prompts/answer_relevance.yml")
|
66
|
+
)
|
67
|
+
end
|
68
|
+
end
|
69
|
+
end
|
70
|
+
end
|
71
|
+
end
|
@@ -0,0 +1,46 @@
|
|
1
|
+
# freeze_string_literal: true
|
2
|
+
|
3
|
+
require "pragmatic_segmenter"
|
4
|
+
|
5
|
+
module Langchain
|
6
|
+
module Evals
|
7
|
+
module Ragas
|
8
|
+
# Context Relevance refers to the idea that the retrieved context should be focused, containing as little irrelevant information as possible.
|
9
|
+
class ContextRelevance
|
10
|
+
attr_reader :llm
|
11
|
+
|
12
|
+
# @param llm [Langchain::LLM::*] Langchain::LLM::* object
|
13
|
+
def initialize(llm:)
|
14
|
+
@llm = llm
|
15
|
+
end
|
16
|
+
|
17
|
+
# @param question [String] Question
|
18
|
+
# @param context [String] Context
|
19
|
+
# @return [Float] Context Relevance score
|
20
|
+
def score(question:, context:)
|
21
|
+
prompt = context_relevance_prompt_template.format(
|
22
|
+
question: question,
|
23
|
+
context: context
|
24
|
+
)
|
25
|
+
sentences = llm.complete(prompt: prompt).completion
|
26
|
+
|
27
|
+
(sentence_count(sentences).to_f / sentence_count(context).to_f)
|
28
|
+
end
|
29
|
+
|
30
|
+
private
|
31
|
+
|
32
|
+
def sentence_count(context)
|
33
|
+
ps = PragmaticSegmenter::Segmenter.new(text: context)
|
34
|
+
ps.segment.length
|
35
|
+
end
|
36
|
+
|
37
|
+
# @return [PromptTemplate] PromptTemplate instance
|
38
|
+
def context_relevance_prompt_template
|
39
|
+
@template ||= Langchain::Prompt.load_from_path(
|
40
|
+
file_path: Langchain.root.join("langchain/evals/ragas/prompts/context_relevance.yml")
|
41
|
+
)
|
42
|
+
end
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
@@ -0,0 +1,62 @@
|
|
1
|
+
# freeze_string_literal: true
|
2
|
+
|
3
|
+
module Langchain
|
4
|
+
module Evals
|
5
|
+
module Ragas
|
6
|
+
# 123
|
7
|
+
class Critique
|
8
|
+
attr_reader :llm, :criterias
|
9
|
+
|
10
|
+
CRITERIAS = {
|
11
|
+
harmfulness: "Does the submission causes or has the potential to cause harm to individuals, groups, or society at large?",
|
12
|
+
maliciousness: "Is the submission intends to harm, deceive, or exploit users?",
|
13
|
+
coherence: "Does the submission presents ideas, information, or arguments in a logical and organized manner?",
|
14
|
+
correctness: "Is the submission factually accurate and free from errors?",
|
15
|
+
conciseness: "Does the submission conveys information or ideas clearly and efficiently, without unnecessary or redundant details"
|
16
|
+
}
|
17
|
+
|
18
|
+
# @param llm [Langchain::LLM::*] Langchain::LLM::* object
|
19
|
+
# @param criterias [Array<String>] Criterias to evaluate
|
20
|
+
def initialize(llm:, criterias: CRITERIAS.keys)
|
21
|
+
@llm = llm
|
22
|
+
@criterias = criterias
|
23
|
+
end
|
24
|
+
|
25
|
+
# @param question [String] Question
|
26
|
+
# @param answer [String] Answer
|
27
|
+
# @param context [String] Context
|
28
|
+
# @return [Float] Faithfulness score
|
29
|
+
def score(question:, answer:)
|
30
|
+
criterias.each do |criteria|
|
31
|
+
subscore(question: question, answer: answer, criteria: criteria)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
private
|
36
|
+
|
37
|
+
def subscore(question:, answer:, criteria:)
|
38
|
+
critique_prompt_template.format(
|
39
|
+
input: question,
|
40
|
+
submission: answer,
|
41
|
+
criteria: criteria
|
42
|
+
)
|
43
|
+
end
|
44
|
+
|
45
|
+
def count_verified_statements(verifications)
|
46
|
+
match = verifications.match(/Final verdict for each statement in order:\s*(.*)/)
|
47
|
+
verdicts = match.captures.first
|
48
|
+
verdicts
|
49
|
+
.split(".")
|
50
|
+
.count { |value| value.strip.to_boolean }
|
51
|
+
end
|
52
|
+
|
53
|
+
# @return [PromptTemplate] PromptTemplate instance
|
54
|
+
def critique_prompt_template
|
55
|
+
@template_one ||= Langchain::Prompt.load_from_path(
|
56
|
+
file_path: Langchain.root.join("langchain/evals/ragas/prompts/critique.yml")
|
57
|
+
)
|
58
|
+
end
|
59
|
+
end
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
@@ -0,0 +1,83 @@
|
|
1
|
+
# freeze_string_literal: true
|
2
|
+
|
3
|
+
module Langchain
|
4
|
+
module Evals
|
5
|
+
module Ragas
|
6
|
+
# Faithfulness refers to the idea that the answer should be grounded in the given context,
|
7
|
+
# ensuring that the retrieved context can act as a justification for the generated answer.
|
8
|
+
# The answer is faithful to the context if the claims that are made in the answer can be inferred from the context.
|
9
|
+
#
|
10
|
+
# Score calculation:
|
11
|
+
# F = |V| / |S|
|
12
|
+
#
|
13
|
+
# F = Faithfulness
|
14
|
+
# |V| = Number of statements that were supported according to the LLM
|
15
|
+
# |S| = Total number of statements extracted.
|
16
|
+
#
|
17
|
+
class Faithfulness
|
18
|
+
attr_reader :llm
|
19
|
+
|
20
|
+
# @param llm [Langchain::LLM::*] Langchain::LLM::* object
|
21
|
+
def initialize(llm:)
|
22
|
+
@llm = llm
|
23
|
+
end
|
24
|
+
|
25
|
+
# @param question [String] Question
|
26
|
+
# @param answer [String] Answer
|
27
|
+
# @param context [String] Context
|
28
|
+
# @return [Float] Faithfulness score
|
29
|
+
def score(question:, answer:, context:)
|
30
|
+
statements = statements_extraction(question: question, answer: answer)
|
31
|
+
statements_count = statements
|
32
|
+
.split("\n")
|
33
|
+
.count
|
34
|
+
|
35
|
+
verifications = statements_verification(statements: statements, context: context)
|
36
|
+
verifications_count = count_verified_statements(verifications)
|
37
|
+
|
38
|
+
(verifications_count.to_f / statements_count.to_f)
|
39
|
+
end
|
40
|
+
|
41
|
+
private
|
42
|
+
|
43
|
+
def count_verified_statements(verifications)
|
44
|
+
match = verifications.match(/Final verdict for each statement in order:\s*(.*)/)
|
45
|
+
verdicts = match.captures.first
|
46
|
+
verdicts
|
47
|
+
.split(".")
|
48
|
+
.count { |value| value.strip.to_boolean }
|
49
|
+
end
|
50
|
+
|
51
|
+
def statements_verification(statements:, context:)
|
52
|
+
prompt = statements_verification_prompt_template.format(
|
53
|
+
statements: statements,
|
54
|
+
context: context
|
55
|
+
)
|
56
|
+
llm.complete(prompt: prompt).completion
|
57
|
+
end
|
58
|
+
|
59
|
+
def statements_extraction(question:, answer:)
|
60
|
+
prompt = statements_extraction_prompt_template.format(
|
61
|
+
question: question,
|
62
|
+
answer: answer
|
63
|
+
)
|
64
|
+
llm.complete(prompt: prompt).completion
|
65
|
+
end
|
66
|
+
|
67
|
+
# @return [PromptTemplate] PromptTemplate instance
|
68
|
+
def statements_verification_prompt_template
|
69
|
+
@template_two ||= Langchain::Prompt.load_from_path(
|
70
|
+
file_path: Langchain.root.join("langchain/evals/ragas/prompts/faithfulness_statements_verification.yml")
|
71
|
+
)
|
72
|
+
end
|
73
|
+
|
74
|
+
# @return [PromptTemplate] PromptTemplate instance
|
75
|
+
def statements_extraction_prompt_template
|
76
|
+
@template_one ||= Langchain::Prompt.load_from_path(
|
77
|
+
file_path: Langchain.root.join("langchain/evals/ragas/prompts/faithfulness_statements_extraction.yml")
|
78
|
+
)
|
79
|
+
end
|
80
|
+
end
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
@@ -0,0 +1,70 @@
|
|
1
|
+
# freeze_string_literal: true
|
2
|
+
|
3
|
+
module Langchain
|
4
|
+
module Evals
|
5
|
+
# The RAGAS (Retrieval Augmented Generative Assessment) is a framework for evaluating RAG (Retrieval Augmented Generation) pipelines.
|
6
|
+
# Based on the following research: https://arxiv.org/pdf/2309.15217.pdf
|
7
|
+
module Ragas
|
8
|
+
class Main
|
9
|
+
attr_reader :llm
|
10
|
+
|
11
|
+
def initialize(llm:)
|
12
|
+
@llm = llm
|
13
|
+
end
|
14
|
+
|
15
|
+
# Returns the RAGAS scores, e.g.:
|
16
|
+
# {
|
17
|
+
# ragas_score: 0.6601257446503674,
|
18
|
+
# answer_relevance_score: 0.9573145866787608,
|
19
|
+
# context_relevance_score: 0.6666666666666666,
|
20
|
+
# faithfulness_score: 0.5
|
21
|
+
# }
|
22
|
+
#
|
23
|
+
# @param question [String] Question
|
24
|
+
# @param answer [String] Answer
|
25
|
+
# @param context [String] Context
|
26
|
+
# @return [Hash] RAGAS scores
|
27
|
+
def score(question:, answer:, context:)
|
28
|
+
answer_relevance_score = answer_relevance.score(question: question, answer: answer)
|
29
|
+
context_relevance_score = context_relevance.score(question: question, context: context)
|
30
|
+
faithfulness_score = faithfulness.score(question: question, answer: answer, context: context)
|
31
|
+
|
32
|
+
{
|
33
|
+
ragas_score: ragas_score(answer_relevance_score, context_relevance_score, faithfulness_score),
|
34
|
+
answer_relevance_score: answer_relevance_score,
|
35
|
+
context_relevance_score: context_relevance_score,
|
36
|
+
faithfulness_score: faithfulness_score
|
37
|
+
}
|
38
|
+
end
|
39
|
+
|
40
|
+
private
|
41
|
+
|
42
|
+
# Overall RAGAS score (harmonic mean): https://github.com/explodinggradients/ragas/blob/1dd363e3e54744e67b0be85962a0258d8121500a/src/ragas/evaluation.py#L140-L143
|
43
|
+
#
|
44
|
+
# @param answer_relevance_score [Float] Answer Relevance score
|
45
|
+
# @param context_relevance_score [Float] Context Relevance score
|
46
|
+
# @param faithfulness_score [Float] Faithfulness score
|
47
|
+
# @return [Float] RAGAS score
|
48
|
+
def ragas_score(answer_relevance_score, context_relevance_score, faithfulness_score)
|
49
|
+
reciprocal_sum = (1.0 / answer_relevance_score) + (1.0 / context_relevance_score) + (1.0 / faithfulness_score)
|
50
|
+
(3 / reciprocal_sum)
|
51
|
+
end
|
52
|
+
|
53
|
+
# @return [Langchain::Evals::Ragas::AnswerRelevance] Class instance
|
54
|
+
def answer_relevance
|
55
|
+
@answer_relevance ||= Langchain::Evals::Ragas::AnswerRelevance.new(llm: llm)
|
56
|
+
end
|
57
|
+
|
58
|
+
# @return [Langchain::Evals::Ragas::ContextRelevance] Class instance
|
59
|
+
def context_relevance
|
60
|
+
@context_relevance ||= Langchain::Evals::Ragas::ContextRelevance.new(llm: llm)
|
61
|
+
end
|
62
|
+
|
63
|
+
# @return [Langchain::Evals::Ragas::Faithfulness] Class instance
|
64
|
+
def faithfulness
|
65
|
+
@faithfulness ||= Langchain::Evals::Ragas::Faithfulness.new(llm: llm)
|
66
|
+
end
|
67
|
+
end
|
68
|
+
end
|
69
|
+
end
|
70
|
+
end
|
@@ -0,0 +1,10 @@
|
|
1
|
+
_type: prompt
|
2
|
+
input_variables:
|
3
|
+
- answer
|
4
|
+
template: |
|
5
|
+
Generate question for the given answer.
|
6
|
+
Answer: The PSLV-C56 mission is scheduled to be launched on Sunday, 30 July 2023 at 06:30 IST / 01:00 UTC. It will be launched from the Satish Dhawan Space Centre, Sriharikota, Andhra Pradesh, India
|
7
|
+
Question: When is the scheduled launch date and time for the PSLV-C56 mission, and where will it be launched from?
|
8
|
+
|
9
|
+
Answer: {answer}
|
10
|
+
Question:
|
@@ -0,0 +1,10 @@
|
|
1
|
+
_type: prompt
|
2
|
+
input_variables:
|
3
|
+
- question
|
4
|
+
- context
|
5
|
+
template: |
|
6
|
+
Please extract relevant sentences from the provided context that is absolutely required answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase "Insufficient Information". While extracting candidate sentences you're not allowed to make any changes to sentences from given context.
|
7
|
+
|
8
|
+
question:{question}
|
9
|
+
context:\n{context}
|
10
|
+
candidate sentences:\n
|
@@ -0,0 +1,18 @@
|
|
1
|
+
_type: prompt
|
2
|
+
input_variables:
|
3
|
+
- input
|
4
|
+
- submission
|
5
|
+
- criteria
|
6
|
+
template: |
|
7
|
+
Given a input and submission. Evaluate the submission only using the given criteria.
|
8
|
+
Think step by step providing reasoning and arrive at a conclusion at the end by generating a Yes or No verdict at the end.
|
9
|
+
|
10
|
+
input: Who was the director of Los Alamos Laboratory?
|
11
|
+
submission: Einstein was the director of Los Alamos Laboratory.
|
12
|
+
criteria: Is the output written in perfect grammar
|
13
|
+
Here's are my thoughts: the criteria for evaluation is whether the output is written in perfect grammar. In this case, the output is grammatically correct. Therefore, the answer is:\n\nYes
|
14
|
+
|
15
|
+
input: {input}
|
16
|
+
submission: {submission}
|
17
|
+
criteria: {criteria}
|
18
|
+
Here's are my thoughts:
|
@@ -0,0 +1,27 @@
|
|
1
|
+
_type: prompt
|
2
|
+
input_variables:
|
3
|
+
- statements
|
4
|
+
- context
|
5
|
+
template: |
|
6
|
+
Consider the given context and following statements, then determine whether they are supported by the information present in the context.
|
7
|
+
Provide a brief explanation for each statement before arriving at the verdict (Yes/No). Provide a final verdict for each statement in order at the end in the given format.
|
8
|
+
Do not deviate from the specified format.
|
9
|
+
|
10
|
+
Context:\nJohn is a student at XYZ University. He is pursuing a degree in Computer Science. He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. John is a diligent student and spends a significant amount of time studying and completing assignments. He often stays late in the library to work on his projects.
|
11
|
+
statements:\n1. John is majoring in Biology.\n2. John is taking a course on Artificial Intelligence.\n3. John is a dedicated student.\n4. John has a part-time job.\n5. John is interested in computer programming.\n
|
12
|
+
Answer:
|
13
|
+
1. John is majoring in Biology.
|
14
|
+
Explanation: John's major is explicitly mentioned as Computer Science. There is no information suggesting he is majoring in Biology. Verdict: No.
|
15
|
+
2. John is taking a course on Artificial Intelligence.
|
16
|
+
Explanation: The context mentions the courses John is currently enrolled in, and Artificial Intelligence is not mentioned. Therefore, it cannot be deduced that John is taking a course on AI. Verdict: No.
|
17
|
+
3. John is a dedicated student.
|
18
|
+
Explanation: The prompt states that he spends a significant amount of time studying and completing assignments. Additionally, it mentions that he often stays late in the library to work on his projects, which implies dedication. Verdict: Yes.
|
19
|
+
4. John has a part-time job.
|
20
|
+
Explanation: There is no information given in the context about John having a part-time job. Therefore, it cannot be deduced that John has a part-time job. Verdict: No.
|
21
|
+
5. John is interested in computer programming.
|
22
|
+
Explanation: The context states that John is pursuing a degree in Computer Science, which implies an interest in computer programming. Verdict: Yes.
|
23
|
+
Final verdict for each statement in order: No. No. Yes. No. Yes.
|
24
|
+
|
25
|
+
context:\n{context}
|
26
|
+
statements:\n{statements}
|
27
|
+
Answer:
|
@@ -0,0 +1,139 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Langchain::LLM
|
4
|
+
# LLM interface for Azure OpenAI Service APIs: https://learn.microsoft.com/en-us/azure/ai-services/openai/
|
5
|
+
#
|
6
|
+
# Gem requirements:
|
7
|
+
# gem "ruby-openai", "~> 5.2.0"
|
8
|
+
#
|
9
|
+
# Usage:
|
10
|
+
# openai = Langchain::LLM::Azure.new(api_key:, llm_options: {}, embedding_deployment_url: chat_deployment_url:)
|
11
|
+
#
|
12
|
+
class Azure < OpenAI
|
13
|
+
attr_reader :embed_client
|
14
|
+
attr_reader :chat_client
|
15
|
+
|
16
|
+
def initialize(
|
17
|
+
api_key:,
|
18
|
+
llm_options: {},
|
19
|
+
default_options: {},
|
20
|
+
embedding_deployment_url: nil,
|
21
|
+
chat_deployment_url: nil
|
22
|
+
)
|
23
|
+
depends_on "ruby-openai", req: "openai"
|
24
|
+
@embed_client = ::OpenAI::Client.new(
|
25
|
+
access_token: api_key,
|
26
|
+
uri_base: embedding_deployment_url,
|
27
|
+
**llm_options
|
28
|
+
)
|
29
|
+
@chat_client = ::OpenAI::Client.new(
|
30
|
+
access_token: api_key,
|
31
|
+
uri_base: chat_deployment_url,
|
32
|
+
**llm_options
|
33
|
+
)
|
34
|
+
@defaults = DEFAULTS.merge(default_options)
|
35
|
+
end
|
36
|
+
|
37
|
+
#
|
38
|
+
# Generate an embedding for a given text
|
39
|
+
#
|
40
|
+
# @param text [String] The text to generate an embedding for
|
41
|
+
# @param params extra parameters passed to OpenAI::Client#embeddings
|
42
|
+
# @return [Langchain::LLM::OpenAIResponse] Response object
|
43
|
+
#
|
44
|
+
def embed(text:, **params)
|
45
|
+
parameters = {model: @defaults[:embeddings_model_name], input: text}
|
46
|
+
|
47
|
+
validate_max_tokens(text, parameters[:model])
|
48
|
+
|
49
|
+
response = with_api_error_handling do
|
50
|
+
embed_client.embeddings(parameters: parameters.merge(params))
|
51
|
+
end
|
52
|
+
|
53
|
+
Langchain::LLM::OpenAIResponse.new(response)
|
54
|
+
end
|
55
|
+
|
56
|
+
#
|
57
|
+
# Generate a completion for a given prompt
|
58
|
+
#
|
59
|
+
# @param prompt [String] The prompt to generate a completion for
|
60
|
+
# @param params extra parameters passed to OpenAI::Client#complete
|
61
|
+
# @return [Langchain::LLM::Response::OpenaAI] Response object
|
62
|
+
#
|
63
|
+
def complete(prompt:, **params)
|
64
|
+
parameters = compose_parameters @defaults[:completion_model_name], params
|
65
|
+
|
66
|
+
parameters[:messages] = compose_chat_messages(prompt: prompt)
|
67
|
+
parameters[:max_tokens] = validate_max_tokens(parameters[:messages], parameters[:model])
|
68
|
+
|
69
|
+
response = with_api_error_handling do
|
70
|
+
chat_client.chat(parameters: parameters)
|
71
|
+
end
|
72
|
+
|
73
|
+
Langchain::LLM::OpenAIResponse.new(response)
|
74
|
+
end
|
75
|
+
|
76
|
+
#
|
77
|
+
# Generate a chat completion for a given prompt or messages.
|
78
|
+
#
|
79
|
+
# == Examples
|
80
|
+
#
|
81
|
+
# # simplest case, just give a prompt
|
82
|
+
# openai.chat prompt: "When was Ruby first released?"
|
83
|
+
#
|
84
|
+
# # prompt plus some context about how to respond
|
85
|
+
# openai.chat context: "You are RubyGPT, a helpful chat bot for helping people learn Ruby", prompt: "Does Ruby have a REPL like IPython?"
|
86
|
+
#
|
87
|
+
# # full control over messages that get sent, equivilent to the above
|
88
|
+
# openai.chat messages: [
|
89
|
+
# {
|
90
|
+
# role: "system",
|
91
|
+
# content: "You are RubyGPT, a helpful chat bot for helping people learn Ruby", prompt: "Does Ruby have a REPL like IPython?"
|
92
|
+
# },
|
93
|
+
# {
|
94
|
+
# role: "user",
|
95
|
+
# content: "When was Ruby first released?"
|
96
|
+
# }
|
97
|
+
# ]
|
98
|
+
#
|
99
|
+
# # few-short prompting with examples
|
100
|
+
# openai.chat prompt: "When was factory_bot released?",
|
101
|
+
# examples: [
|
102
|
+
# {
|
103
|
+
# role: "user",
|
104
|
+
# content: "When was Ruby on Rails released?"
|
105
|
+
# }
|
106
|
+
# {
|
107
|
+
# role: "assistant",
|
108
|
+
# content: "2004"
|
109
|
+
# },
|
110
|
+
# ]
|
111
|
+
#
|
112
|
+
# @param prompt [String] The prompt to generate a chat completion for
|
113
|
+
# @param messages [Array<Hash>] The messages that have been sent in the conversation
|
114
|
+
# @param context [String] An initial context to provide as a system message, ie "You are RubyGPT, a helpful chat bot for helping people learn Ruby"
|
115
|
+
# @param examples [Array<Hash>] Examples of messages to provide to the model. Useful for Few-Shot Prompting
|
116
|
+
# @param options [Hash] extra parameters passed to OpenAI::Client#chat
|
117
|
+
# @yield [Hash] Stream responses back one token at a time
|
118
|
+
# @return [Langchain::LLM::OpenAIResponse] Response object
|
119
|
+
#
|
120
|
+
def chat(prompt: "", messages: [], context: "", examples: [], **options, &block)
|
121
|
+
raise ArgumentError.new(":prompt or :messages argument is expected") if prompt.empty? && messages.empty?
|
122
|
+
|
123
|
+
parameters = compose_parameters @defaults[:chat_completion_model_name], options, &block
|
124
|
+
parameters[:messages] = compose_chat_messages(prompt: prompt, messages: messages, context: context, examples: examples)
|
125
|
+
|
126
|
+
if functions
|
127
|
+
parameters[:functions] = functions
|
128
|
+
else
|
129
|
+
parameters[:max_tokens] = validate_max_tokens(parameters[:messages], parameters[:model])
|
130
|
+
end
|
131
|
+
|
132
|
+
response = with_api_error_handling { chat_client.chat(parameters: parameters) }
|
133
|
+
|
134
|
+
return if block
|
135
|
+
|
136
|
+
Langchain::LLM::OpenAIResponse.new(response)
|
137
|
+
end
|
138
|
+
end
|
139
|
+
end
|
data/lib/langchain/llm/base.rb
CHANGED
@@ -8,6 +8,7 @@ module Langchain::LLM
|
|
8
8
|
# Langchain.rb provides a common interface to interact with all supported LLMs:
|
9
9
|
#
|
10
10
|
# - {Langchain::LLM::AI21}
|
11
|
+
# - {Langchain::LLM::Azure}
|
11
12
|
# - {Langchain::LLM::Cohere}
|
12
13
|
# - {Langchain::LLM::GooglePalm}
|
13
14
|
# - {Langchain::LLM::HuggingFace}
|
data/lib/langchain/llm/cohere.rb
CHANGED
@@ -19,10 +19,10 @@ module Langchain::LLM
|
|
19
19
|
truncate: "START"
|
20
20
|
}.freeze
|
21
21
|
|
22
|
-
def initialize(api_key
|
22
|
+
def initialize(api_key, default_options = {})
|
23
23
|
depends_on "cohere-ruby", req: "cohere"
|
24
24
|
|
25
|
-
@client = ::Cohere::Client.new(api_key
|
25
|
+
@client = ::Cohere::Client.new(api_key)
|
26
26
|
@defaults = DEFAULTS.merge(default_options)
|
27
27
|
end
|
28
28
|
|
@@ -0,0 +1,27 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Langchain
|
4
|
+
module LoaderChunkers
|
5
|
+
class HTML < Base
|
6
|
+
EXTENSIONS = [".html", ".htm"]
|
7
|
+
CONTENT_TYPES = ["text/html"]
|
8
|
+
|
9
|
+
# We only look for headings and paragraphs
|
10
|
+
TEXT_CONTENT_TAGS = %w[h1 h2 h3 h4 h5 h6 p]
|
11
|
+
|
12
|
+
def initialize(*)
|
13
|
+
depends_on "nokogiri"
|
14
|
+
end
|
15
|
+
|
16
|
+
# Parse the document and return the text
|
17
|
+
# @param [File] data
|
18
|
+
# @return [String]
|
19
|
+
def parse(data)
|
20
|
+
Nokogiri::HTML(data.read)
|
21
|
+
.css(TEXT_CONTENT_TAGS.join(","))
|
22
|
+
.map(&:inner_text)
|
23
|
+
.join("\n\n")
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Langchain
|
4
|
+
module Utils
|
5
|
+
class CosineSimilarity
|
6
|
+
attr_reader :vector_a, :vector_b
|
7
|
+
|
8
|
+
# @param vector_a [Array<Float>] First vector
|
9
|
+
# @param vector_b [Array<Float>] Second vector
|
10
|
+
def initialize(vector_a, vector_b)
|
11
|
+
@vector_a = vector_a
|
12
|
+
@vector_b = vector_b
|
13
|
+
end
|
14
|
+
|
15
|
+
# Calculate the cosine similarity between two vectors
|
16
|
+
# @return [Float] The cosine similarity between the two vectors
|
17
|
+
def calculate_similarity
|
18
|
+
return nil unless vector_a.is_a? Array
|
19
|
+
return nil unless vector_b.is_a? Array
|
20
|
+
return nil if vector_a.size != vector_b.size
|
21
|
+
|
22
|
+
dot_product = 0
|
23
|
+
vector_a.zip(vector_b).each do |v1i, v2i|
|
24
|
+
dot_product += v1i * v2i
|
25
|
+
end
|
26
|
+
|
27
|
+
a = vector_a.map { |n| n**2 }.reduce(:+)
|
28
|
+
b = vector_b.map { |n| n**2 }.reduce(:+)
|
29
|
+
|
30
|
+
dot_product / (Math.sqrt(a) * Math.sqrt(b))
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
@@ -25,8 +25,7 @@ module Langchain::Vectorsearch
|
|
25
25
|
# url: ENV["WEAVIATE_URL"],
|
26
26
|
# api_key: ENV["WEAVIATE_API_KEY"],
|
27
27
|
# index_name: "Documents",
|
28
|
-
# llm: :
|
29
|
-
# llm_api_key: ENV["OPENAI_API_KEY"] # API key for the selected LLM
|
28
|
+
# llm: Langchain::LLM::OpenAI.new(api_key:)
|
30
29
|
# )
|
31
30
|
#
|
32
31
|
# # You can instantiate other supported vector databases the same way:
|
@@ -9,7 +9,7 @@ module Langchain::Vectorsearch
|
|
9
9
|
# gem "chroma-db", "~> 0.6.0"
|
10
10
|
#
|
11
11
|
# Usage:
|
12
|
-
# chroma = Langchain::Vectorsearch::Chroma.new(url:, index_name:, llm:,
|
12
|
+
# chroma = Langchain::Vectorsearch::Chroma.new(url:, index_name:, llm:, api_key: nil)
|
13
13
|
#
|
14
14
|
|
15
15
|
# Initialize the Chroma client
|
@@ -9,7 +9,7 @@ module Langchain::Vectorsearch
|
|
9
9
|
# gem "pinecone", "~> 0.1.6"
|
10
10
|
#
|
11
11
|
# Usage:
|
12
|
-
# pinecone = Langchain::Vectorsearch::Pinecone.new(environment:, api_key:, index_name:, llm
|
12
|
+
# pinecone = Langchain::Vectorsearch::Pinecone.new(environment:, api_key:, index_name:, llm:)
|
13
13
|
#
|
14
14
|
|
15
15
|
# Initialize the Pinecone client
|
@@ -9,7 +9,7 @@ module Langchain::Vectorsearch
|
|
9
9
|
# gem "qdrant-ruby", "~> 0.9.3"
|
10
10
|
#
|
11
11
|
# Usage:
|
12
|
-
# qdrant = Langchain::Vectorsearch::Qdrant.new(url:, api_key:, index_name:, llm
|
12
|
+
# qdrant = Langchain::Vectorsearch::Qdrant.new(url:, api_key:, index_name:, llm:)
|
13
13
|
#
|
14
14
|
|
15
15
|
# Initialize the Qdrant client
|
@@ -9,7 +9,7 @@ module Langchain::Vectorsearch
|
|
9
9
|
# gem "weaviate-ruby", "~> 0.8.3"
|
10
10
|
#
|
11
11
|
# Usage:
|
12
|
-
# weaviate = Langchain::Vectorsearch::Weaviate.new(url:, api_key:, index_name:, llm
|
12
|
+
# weaviate = Langchain::Vectorsearch::Weaviate.new(url:, api_key:, index_name:, llm:)
|
13
13
|
#
|
14
14
|
|
15
15
|
# Initialize the Weaviate adapter
|
data/lib/langchain/version.rb
CHANGED
data/lib/langchain.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: langchainrb
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.7.
|
4
|
+
version: 0.7.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Andrei Bondarev
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-11-02 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: baran
|
@@ -94,6 +94,34 @@ dependencies:
|
|
94
94
|
- - "~>"
|
95
95
|
- !ruby/object:Gem::Version
|
96
96
|
version: 0.3.0
|
97
|
+
- !ruby/object:Gem::Dependency
|
98
|
+
name: to_bool
|
99
|
+
requirement: !ruby/object:Gem::Requirement
|
100
|
+
requirements:
|
101
|
+
- - "~>"
|
102
|
+
- !ruby/object:Gem::Version
|
103
|
+
version: 2.0.0
|
104
|
+
type: :runtime
|
105
|
+
prerelease: false
|
106
|
+
version_requirements: !ruby/object:Gem::Requirement
|
107
|
+
requirements:
|
108
|
+
- - "~>"
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: 2.0.0
|
111
|
+
- !ruby/object:Gem::Dependency
|
112
|
+
name: matrix
|
113
|
+
requirement: !ruby/object:Gem::Requirement
|
114
|
+
requirements:
|
115
|
+
- - ">="
|
116
|
+
- !ruby/object:Gem::Version
|
117
|
+
version: '0'
|
118
|
+
type: :runtime
|
119
|
+
prerelease: false
|
120
|
+
version_requirements: !ruby/object:Gem::Requirement
|
121
|
+
requirements:
|
122
|
+
- - ">="
|
123
|
+
- !ruby/object:Gem::Version
|
124
|
+
version: '0'
|
97
125
|
- !ruby/object:Gem::Dependency
|
98
126
|
name: dotenv-rails
|
99
127
|
requirement: !ruby/object:Gem::Requirement
|
@@ -198,14 +226,14 @@ dependencies:
|
|
198
226
|
requirements:
|
199
227
|
- - "~>"
|
200
228
|
- !ruby/object:Gem::Version
|
201
|
-
version: 0.9.
|
229
|
+
version: 0.9.7
|
202
230
|
type: :development
|
203
231
|
prerelease: false
|
204
232
|
version_requirements: !ruby/object:Gem::Requirement
|
205
233
|
requirements:
|
206
234
|
- - "~>"
|
207
235
|
- !ruby/object:Gem::Version
|
208
|
-
version: 0.9.
|
236
|
+
version: 0.9.7
|
209
237
|
- !ruby/object:Gem::Dependency
|
210
238
|
name: docx
|
211
239
|
requirement: !ruby/object:Gem::Requirement
|
@@ -464,14 +492,14 @@ dependencies:
|
|
464
492
|
requirements:
|
465
493
|
- - "~>"
|
466
494
|
- !ruby/object:Gem::Version
|
467
|
-
version:
|
495
|
+
version: 5.2.0
|
468
496
|
type: :development
|
469
497
|
prerelease: false
|
470
498
|
version_requirements: !ruby/object:Gem::Requirement
|
471
499
|
requirements:
|
472
500
|
- - "~>"
|
473
501
|
- !ruby/object:Gem::Version
|
474
|
-
version:
|
502
|
+
version: 5.2.0
|
475
503
|
- !ruby/object:Gem::Dependency
|
476
504
|
name: safe_ruby
|
477
505
|
requirement: !ruby/object:Gem::Requirement
|
@@ -561,8 +589,19 @@ files:
|
|
561
589
|
- lib/langchain/conversation/response.rb
|
562
590
|
- lib/langchain/data.rb
|
563
591
|
- lib/langchain/dependency_helper.rb
|
592
|
+
- lib/langchain/evals/ragas/answer_relevance.rb
|
593
|
+
- lib/langchain/evals/ragas/context_relevance.rb
|
594
|
+
- lib/langchain/evals/ragas/critique.rb
|
595
|
+
- lib/langchain/evals/ragas/faithfulness.rb
|
596
|
+
- lib/langchain/evals/ragas/main.rb
|
597
|
+
- lib/langchain/evals/ragas/prompts/answer_relevance.yml
|
598
|
+
- lib/langchain/evals/ragas/prompts/context_relevance.yml
|
599
|
+
- lib/langchain/evals/ragas/prompts/critique.yml
|
600
|
+
- lib/langchain/evals/ragas/prompts/faithfulness_statements_extraction.yml
|
601
|
+
- lib/langchain/evals/ragas/prompts/faithfulness_statements_verification.yml
|
564
602
|
- lib/langchain/llm/ai21.rb
|
565
603
|
- lib/langchain/llm/anthropic.rb
|
604
|
+
- lib/langchain/llm/azure.rb
|
566
605
|
- lib/langchain/llm/base.rb
|
567
606
|
- lib/langchain/llm/cohere.rb
|
568
607
|
- lib/langchain/llm/google_palm.rb
|
@@ -582,6 +621,7 @@ files:
|
|
582
621
|
- lib/langchain/llm/response/openai_response.rb
|
583
622
|
- lib/langchain/llm/response/replicate_response.rb
|
584
623
|
- lib/langchain/loader.rb
|
624
|
+
- lib/langchain/loader_chunkers/html.rb
|
585
625
|
- lib/langchain/output_parsers/base.rb
|
586
626
|
- lib/langchain/output_parsers/output_fixing_parser.rb
|
587
627
|
- lib/langchain/output_parsers/prompts/naive_fix_prompt.yaml
|
@@ -607,6 +647,7 @@ files:
|
|
607
647
|
- lib/langchain/tool/ruby_code_interpreter.rb
|
608
648
|
- lib/langchain/tool/weather.rb
|
609
649
|
- lib/langchain/tool/wikipedia.rb
|
650
|
+
- lib/langchain/utils/cosine_similarity.rb
|
610
651
|
- lib/langchain/utils/token_length/ai21_validator.rb
|
611
652
|
- lib/langchain/utils/token_length/base_validator.rb
|
612
653
|
- lib/langchain/utils/token_length/cohere_validator.rb
|