rspec-llama 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 54cccc6da1341c23e2ffc84c43095c011e9c6c58255854886ba85ccddb274e67
+  data.tar.gz: 43e06cd643a0755eac0dfe3ce2048153cd57fa9c2f57029fe3285e650032fc2a
+SHA512:
+  metadata.gz: 42b5b576d38336a09aecf115ba4e3fa3a59d17d7f4c5c107c627429f11be0e2079d3f5e9d8523195c8b45d3e351199b4cb0d1826f483c1a59859e0bc3e0b9d6a
+  data.tar.gz: 45ce831f3e2997d6cc4b3953fdd9ea70d3f01fc4bbcfe0d58b5b69d3c16413ae484dfed675caa2e050dfa26feb79448f0c189f925b1279ce608c4386ec90a12c

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at vadser1999@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [https://contributor-covenant.org/version/1/4][version]
+
+[homepage]: https://contributor-covenant.org
+[version]: https://contributor-covenant.org/version/1/4/

data/LICENSE

@@ -0,0 +1,220 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [2024] [AI Foundry]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+
+Dlab Subcomponents:
+
+Dlab project contains subcomponents with separate copyright
+notices and license terms. Your use of the source code for the these
+subcomponents is subject to the terms and conditions of the following
+licenses.
+
+========================================================================
+Apache 2.0 licenses
+========================================================================
+
+The following components are provided under the Apache 2.0 License. See project link for details.
+
+(Apache 2.0 License) Material design icons (https://github.com/google/material-design-icons)
+(Apache 2.0 License) Google Fonts Files, Open Sans (https://github.com/google/fonts/tree/master/apache/opensans)

data/README.md

@@ -0,0 +1,350 @@
+# RSpec::Llama
+
+[![Gem Version](https://badge.fury.io/rb/rspec-llama.svg)](https://badge.fury.io/rb/rspec-llama)
+[![Build Status](https://github.com/aifoundry-org/rspec-llama/actions/workflows/rspec-llama.yml/badge.svg)](https://github.com/aifoundry-org/rspec-llama/actions)
+
+## Introduction
+
+**RSpec::Llama** is a versatile testing framework designed to integrate AI model testing seamlessly into the RSpec ecosystem. Whether you're working with OpenAI's GPT models, Llama, or other AI models, RSpec::Llama simplifies the process of configuring, running, and validating your models' outputs.
+
+## Features
+
+- **Model Configurations**: Easily set up and customize configurations for various AI models like OpenAI, Llama, and Ollama. Configure models with parameters such as temperature, token limits, and stop sequences.
+- **Model Runners**: Seamlessly run AI models using predefined configurations, allowing you to execute prompts and capture their outputs in a simple and consistent way.
+- **Comprehensive Assertions**: Validate model outputs against expected results using advanced matchers, such as `match`, `match_all`, `match_any`, and `match_none`, to ensure your models behave as expected in different scenarios.
+- **RSpec Integration**: Fully integrated with RSpec, enabling AI model testing to fit naturally into your existing test suite.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'rspec-llama'
+```
+
+And then execute:
+
+```
+$ bundle install
+```
+
+Or install it yourself as:
+
+```
+$ gem install rspec-llama
+```
+
+## RSpec Helpers
+
+RSpec::Llama provides a set of helpers to simplify the configuration and interaction with AI models during testing.
+These helpers allow you to easily define model configurations, runners, and prompts, making it easier to integrate AI
+models like OpenAI, Llama, and Ollama into your RSpec tests.
+
+### `build_model_configuration(configuration_type, **options)`
+
+The `build_model_configuration` helper is used to create a model configuration object for various AI models,
+allowing you to specify and customize the parameters that will be used during the model's execution.
+This helper supports different types of model configurations, such as OpenAI's GPT models and Llama models.
+
+Parameters:
+
+`configuration_type`: Symbol representing the type of model configuration. Possible values are `:openai`, `:llama_cpp`, and `:ollama`.
+
+`options`: Hash of configuration options specific to the selected model type.
+
+#### OpenAI Model Configuration
+
+Supported Options:
+
+`model`: The model to use (e.g., 'gpt-3.5-turbo', 'gpt-4').
+
+`temperature`: Sampling temperature between 0 and 2. Higher values make output more random.
+
+`stop`: Up to 4 sequences where the API will stop generating further tokens.
+
+`seed`: A seed for controlling the randomness in generation. This ensures consistent outputs between runs when the same seed and model configuration are used. Useful for debugging and testing.
+
+```ruby
+config = build_model_configuration(:openai, model: 'gpt-4', temperature: 0.7)
+
+# Example usage
+runner = build_model_runner(:openai, access_token: ENV['OPENAI_ACCESS_TOKEN'])
+prompt = 'What gems does the Rails gem depend on?'
+result = runner.call(config, prompt)
+
+expect(result).to match_all(
+  'activesupport', 'activerecord', 'actionpack', 'actionview',
+  'actionmailer', 'actioncable', 'railties'
+)
+```
+
+#### LlamaCpp Model Configuration
+
+The LlamaCpp model configuration is used to set parameters for models running with the llama.cpp implementation.
+
+```ruby
+config = build_model_configuration(:llama_cpp, model: '/path/to/model', temperature: 0.5, predict: 500)
+
+# Example usage
+runner = build_model_runner(:llama_cpp, cli_path: '/path/to/llama-cli')
+prompt = 'What are the most popular Ruby frameworks?'
+result = runner.call(config, prompt)
+
+expect(result).to match_all('Ruby on Rails', 'Sinatra', 'Hanami')
+```
+
+Supported Options:
+
+`model`: The path to the model file.
+
+`temperature`: Sampling temperature between 0 and 2.
+
+`predict`: The number of tokens to predict.
+
+`stop`: Regular expression to define where the model should stop generating text.
+
+#### Ollama Model Configuration
+
+The Ollama model configuration is similar to the OpenAI and LlamaCpp configurations but tailored to the Ollama models.
+
+```ruby
+config = build_model_configuration(:ollama, model: 'ollama3.1')
+
+# Example usage
+runner = build_model_runner(:ollama)
+prompt = 'Who created the Ruby programming language?'
+result = runner.call(config, prompt)
+
+expect(result).to match_all('Yukihiro', 'Matz', 'Matsumoto')
+```
+
+Supported Options:
+
+`model`: The model to use (e.g., 'ollama3.1').
+
+### `build_model_runner(runner_type, **options)`
+
+The `build_model_runner` helper is used to create a model runner object that interacts with various AI models,
+allowing you to execute prompts and retrieve results. This helper supports different types of model runners,
+such as those for OpenAI's GPT models, Llama models, and others.
+
+Parameters:
+
+`runner_type`: Symbol representing the type of model runner. Possible values are `:openai`, `:llama_cpp`, and `:ollama`.
+
+`options`: Hash of options specific to the selected runner type, such as API credentials or executable paths.
+
+#### OpenAI Model Runner
+
+The OpenAI model runner interacts with OpenAI's API to execute prompts and retrieve responses from models like GPT-3.5 or GPT-4.
+
+```ruby
+runner = build_model_runner(:openai, access_token: ENV['OPENAI_ACCESS_TOKEN'], organization_id: ENV['OPENAI_ORGANIZATION_ID'])
+
+# Example usage
+config = build_model_configuration(:openai, model: 'gpt-4', temperature: 0.7)
+prompt = 'What is the capital of France?'
+result = runner.call(config, prompt)
+puts result.to_s
+```
+
+Supported Options:
+
+`access_token`: The API key for accessing OpenAI's API.
+
+`organization_id`: (Optional) The ID of your OpenAI organization.
+
+`project_id`: (Optional) The ID of your OpenAI project.
+
+#### LlamaCpp Model Runner
+
+coming soon
+
+#### Ollama Model Runner
+
+coming soon
+
+## RSpec Matchers
+
+This gem provides RSpec matchers for comparing model outputs, focusing on language models like those
+from OpenAI and Llama. These matchers help assert the presence or absence of specific strings or patterns
+in the output, and they are designed to work seamlessly with RSpec's syntax.
+
+### `match(expected)`
+
+The `match` matcher is used to check if a string or pattern is present in the actual output.
+
+```ruby
+expect(result).to match('Ruby on Rails')
+expect(result).to match(/Rails/)
+```
+
+**Passes if**: The output contains the exact string or matches the given regular expression.
+
+**Fails if**: The string or pattern is not present in the output.
+
+### `match_all(*expected)`
+
+The `match_all` matcher checks if all the provided strings or patterns are present in the actual output.
+
+```ruby
+expect(result).to match_all('Ruby on Rails', 'Sinatra', 'Hanami')
+expect(result).to match_all(/Rails/, /Sinatra/)
+```
+
+**Passes if**: All strings or patterns are found in the output.
+
+**Fails if**: Any one of the strings or patterns is missing from the output.
+
+### `match_any(*expected)`
+
+The `match_any` matcher checks if any of the provided strings or patterns are present in the actual output.
+
+```ruby
+expect(result).to match_any('RoR', 'Ruby on Rails')
+expect(result).to match_any(/Rails/, /Sinatra/)
+```
+
+**Passes if**: At least one string or pattern is found in the output.
+
+**Fails if**: None of the strings or patterns are found in the output.
+
+### `match_none(*expected)`
+
+The `match_none` matcher ensures that none of the provided strings or patterns are present in the actual output.
+
+```ruby
+expect(result).to match_none('Django', 'Flask', 'Symfony')
+expect(result).to match_none(/Django/, /Flask/)
+```
+
+**Passes if**: None of the strings or patterns are found in the output.
+
+**Fails if**: Any one of the strings or patterns is found in the output.
+
+## Full Example
+
+Here’s a full example that demonstrates how to use helpers and matchers to test various models with different configurations.
+
+```ruby
+require 'rspec/llama'
+
+RSpec.shared_examples 'application frameworks' do
+  describe 'popular Ruby frameworks' do
+    let(:prompt) { 'What are the most popular Ruby frameworks?' }
+
+    it 'matches Ruby frameworks', :aggregate_failures do
+      result = run_model!
+
+      expect(result).to match_all('Ruby on Rails', 'Sinatra', 'Hanami')
+      expect(result).to match_none('Django', 'Flask', 'Symfony', 'Laravel', 'Yii')
+    end
+  end
+
+  describe 'dependencies for the Rails gem' do
+    let(:prompt) { 'What gems does the Rails gem depend on?' }
+
+    it 'matches Rails dependencies' do
+      result = run_model!
+
+      expect(result).to match_all(
+        'activesupport', 'activerecord', 'actionpack', 'actionview',
+        'actionmailer', 'actioncable', 'railties'
+      )
+    end
+  end
+
+  describe 'popular Python frameworks' do
+    let(:prompt) { 'What are the most popular Python frameworks?' }
+
+    it 'matches Python frameworks', :aggregate_failures do
+      result = run_model!
+
+      expect(result).to match_all('Django', 'Flask')
+      expect(result).to match_none('Ruby on Rails', 'Sinatra', 'Hanami', 'Symfony', 'Laravel', 'Yii')
+    end
+  end
+end
+
+RSpec.describe 'Popular Application Frameworks' do
+  subject(:run_model!) { runner.call(config, prompt) }
+
+  context 'with OpenAI model runner' do
+    let(:runner) { build_model_runner(:openai, access_token: ENV.fetch('OPENAI_ACCESS_TOKEN')) }
+    let(:config) { build_model_configuration(:openai, model:, temperature:, seed: RSpec.configuration.seed) }
+    let(:temperature) { 0.5 }
+
+    context 'with gpt-4o-mini model' do
+      let(:model) { 'gpt-4o-mini' }
+
+      include_examples 'application frameworks'
+    end
+
+    context 'with gpt-4o model' do
+      let(:model) { 'gpt-4o' }
+
+      include_examples 'application frameworks'
+    end
+
+    context 'with gpt-4-turbo model' do
+      let(:model) { 'gpt-4-turbo' }
+
+      include_examples 'application frameworks'
+
+      context 'with different temperature' do
+        let(:temperature) { 0.1 }
+
+        include_examples 'application frameworks'
+      end
+    end
+  end
+end
+```
+
+## Development
+
+To contribute to the development of this gem, follow the steps below:
+
+### Setting Up the Development Environment
+
+1. Clone the repository:
+
+   ```bash
+   git clone https://github.com/aifoundry-org/rspec-llama.git
+   cd rspec-llama
+   ```
+
+2. Install dependencies:
+
+   Make sure you have Bundler installed. Then run:
+
+   ```bash
+   bundle install
+   ```
+
+3. Run the tests:
+
+   This gem uses RSpec for testing. To run the tests:
+
+   ```bash
+   bundle exec rspec
+   ```
+
+4. Run Rubocop for code linting
+
+   Ensure your code follows the community Ruby style guide by running Rubocop:
+   ```bash
+   bundle exec rubocop
+   ```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/aifoundry-org/rspec-llama. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/aifoundry-org/rspec-llama/blob/master/CODE_OF_CONDUCT.md).
+
+## License
+
+This project is licensed under the Apache License 2.0. See the [LICENSE](LICENSE) file for details.
+
+## Code of Conduct
+
+Everyone interacting in the Rspec::Llama project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/aifoundry-org/rspec-llama/blob/master/CODE_OF_CONDUCT.md).

data/lib/rspec/llama/helpers.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    module Helpers
+      RUNNER_TYPES = {
+        openai: 'OpenaiModelRunner',
+        ollama: 'OllamaModelRunner',
+        llama_cpp: 'LlamaCppModelRunner'
+      }.freeze
+
+      CONFIGURATION_TYPES = {
+        openai: 'OpenaiModelConfiguration',
+        ollama: 'OllamaModelConfiguration',
+        llama_cpp: 'LlamaCppModelConfiguration'
+      }.freeze
+
+      ASSERTION_TYPES = {
+        exclude_all: 'ExcludeAllModelAssertion',
+        include_all: 'IncludeAllModelAssertion',
+        include_any: 'IncludeAnyModelAssertion'
+      }.freeze
+
+      def build_model_configuration(configuration_type, **)
+        configuration_class = CONFIGURATION_TYPES[configuration_type.to_sym]
+        raise "Unsupported model configuration type: #{configuration_type}" unless configuration_class
+
+        RSpec::Llama.const_get(configuration_class).new(**)
+      end
+
+      def build_model_runner(runner_type, **)
+        runner_class = RUNNER_TYPES[runner_type.to_sym]
+        raise "Unsupported model runner type: #{runner_type}" unless runner_class
+
+        RSpec::Llama.const_get(runner_class).new(**)
+      end
+    end
+  end
+end

data/lib/rspec/llama/matchers/match.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    module Matchers
+      class Match < RSpec::Matchers::BuiltIn::Match
+        def matches?(actual)
+          return super unless actual.is_a?(RSpec::Llama::BaseModelRunnerResult)
+
+          @actual = actual
+          @actual.to_s.match?(@expected)
+        end
+
+        def failure_message
+          return super unless @actual.is_a?(RSpec::Llama::BaseModelRunnerResult)
+
+          "expected #{@actual.to_s.inspect} to match #{@expected.inspect}"
+        end
+
+        def failure_message_when_negated
+          return super unless @actual.is_a?(RSpec::Llama::BaseModelRunnerResult)
+
+          "expected #{@actual.to_s.inspect} not to match #{@expected.inspect}"
+        end
+
+        def diffable?
+          return super unless @actual.is_a?(RSpec::Llama::BaseModelRunnerResult)
+
+          false
+        end
+      end
+    end
+  end
+end

data/lib/rspec/llama/matchers/match_all.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    module Matchers
+      class MatchAll
+        def initialize(*expected)
+          @expected = expected
+        end
+
+        def description
+          "match all of #{@expected.inspect}"
+        end
+
+        def matches?(actual)
+          @actual = actual
+          @unmatched_values = @expected.reject { |exp| @actual.to_s.match?(exp) }
+
+          @unmatched_values.empty?
+        end
+
+        def failure_message
+          "expected #{@actual.to_s.inspect} to match all of #{@expected.inspect}, " \
+            "but it did not match: #{@unmatched_values.inspect}"
+        end
+
+        def failure_message_when_negated
+          "expected #{@actual.to_s.inspect} not to match all of #{@expected.inspect}, but it matched all."
+        end
+      end
+    end
+  end
+end

data/lib/rspec/llama/matchers/match_any.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    module Matchers
+      class MatchAny
+        def initialize(*expected)
+          @expected = expected
+        end
+
+        def description
+          "match any of #{@expected.inspect}"
+        end
+
+        def matches?(actual)
+          @actual = actual
+          @unmatched_values = @expected.reject { |exp| @actual.to_s.match?(exp) }
+
+          @unmatched_values.size < @expected.size
+        end
+
+        def failure_message
+          "expected #{@actual.to_s.inspect} to match any of #{@expected.inspect}, but none of them matched."
+        end
+
+        def failure_message_when_negated
+          "expected #{@actual.to_s.inspect} not to match any of #{@expected.inspect}, but it matched some."
+        end
+      end
+    end
+  end
+end

data/lib/rspec/llama/matchers/match_none.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    module Matchers
+      class MatchNone
+        def initialize(*expected)
+          @expected = expected
+        end
+
+        def description
+          "match none of #{@expected.inspect}"
+        end
+
+        def matches?(actual)
+          @actual = actual
+          @matched_values = @expected.select { |exp| @actual.to_s.match?(exp) }
+
+          @matched_values.empty?
+        end
+
+        def failure_message
+          "expected #{@actual.to_s.inspect} to match none of #{@expected.inspect}, " \
+            "but it matched: #{@matched_values.inspect}"
+        end
+
+        def failure_message_when_negated
+          "expected #{@actual.to_s.inspect} to match at least one of #{@expected.inspect}, but none matched."
+        end
+      end
+    end
+  end
+end

data/lib/rspec/llama/matchers.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    module Matchers
+      {
+        Match: 'rspec/llama/matchers/match',
+        MatchAll: 'rspec/llama/matchers/match_all',
+        MatchAny: 'rspec/llama/matchers/match_any',
+        MatchNone: 'rspec/llama/matchers/match_none'
+      }.each { |class_name, path| autoload class_name, path }
+
+      def match(expected)
+        Match.new(expected)
+      end
+
+      def match_all(*expected)
+        MatchAll.new(*expected)
+      end
+
+      def match_any(*expected)
+        MatchAny.new(*expected)
+      end
+
+      def match_none(*expected)
+        MatchNone.new(*expected)
+      end
+    end
+  end
+end

data/lib/rspec/llama/model_configurations/llama_cpp_model_configuration.rb

@@ -0,0 +1,72 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    class LlamaCppModelConfiguration
+      DEFAULT_TEMPERATURE = 0.5
+      DEFAULT_PREDICT = 500
+      DEFAULT_STOP = /\A<(?:end|user|assistant|endoftext|system)>\z/
+
+      attr_reader :model, :temperature, :predict, :stop, :additional_options
+
+      # Initializes a new configuration for the llama.cpp model.
+      #
+      # @param [String] model The path to the model file that will be used.
+      # @param [Float] temperature The temperature for sampling, between 0 and 2. Higher values
+      #   make the output more random, while lower values make it more focused. Defaults to 0.5.
+      # @param [Integer] predict The number of tokens to predict. Defaults to 500.
+      # @param [Regexp] stop The stop token that signals the end of generation. Defaults to a regular expression
+      #   that matches common end-of-text tokens.
+      # @param [Hash] additional_options Additional configuration options, where keys are the option names and values
+      #   are either true (for flags) or values for key-value pairs. These options are passed directly to the CLI.
+      #
+      # @example Basic usage with a specific model path and default parameters
+      #   config = RSpec::Llama::LlamaCppModelConfiguration.new(
+      #     model: '/path/to/model'
+      #   )
+      #
+      # @example Custom parameters and additional CLI options
+      #   config = RSpec::Llama::LlamaCppModelConfiguration.new(
+      #     model: '/path/to/model',
+      #     temperature: 0.7,
+      #     predict: 300,
+      #     threads: 8,
+      #     verbose: true,
+      #     log_file: '/path/to/logfile'
+      #   )
+      def initialize(
+        model:,
+        temperature: DEFAULT_TEMPERATURE,
+        predict: DEFAULT_PREDICT,
+        stop: DEFAULT_STOP,
+        **additional_options
+      )
+        @model = model
+        @temperature = temperature
+        @predict = predict
+        @stop = stop
+        @additional_options = additional_options
+      end
+
+      # Converts the configuration into an array of CLI options.
+      #
+      # @return [Array<String>] An array of strings representing CLI options, where each key-value
+      #   pair or flag is converted to the appropriate format for passing to the llama.cpp executable.
+      #
+      # @example CLI options
+      #   config.to_a
+      #   # => ['--model', '/path/to/model', '--temp', '0.7', '--verbose', '--log-file', '/path/to/logfile']
+      def to_a
+        cli_options = ['--model', model, '--temp', temperature.to_s, '--predict', predict.to_s]
+
+        # Add additional options in key-value pair format
+        additional_options.each do |option, value|
+          cli_options << "--#{option}".tr('_', '-')
+          cli_options << value.to_s unless value == true
+        end
+
+        cli_options
+      end
+    end
+  end
+end

data/lib/rspec/llama/model_configurations/ollama_model_configuration.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    class OllamaModelConfiguration
+      DEFAULT_MODEL = 'llama3.1'
+
+      attr_reader :model
+
+      def initialize(model: DEFAULT_MODEL)
+        @model = model
+      end
+    end
+  end
+end

data/lib/rspec/llama/model_configurations/openai_model_configuration.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    class OpenaiModelConfiguration
+      DEFAULT_MODEL = 'gpt-3.5-turbo'
+      DEFAULT_TEMPERATURE = 0.5
+
+      attr_reader :model, :temperature, :additional_options
+
+      # Initializes a new configuration for the OpenAI model.
+      #
+      # @param [String] model The model to use. Defaults to 'gpt-3.5-turbo'.
+      # @param [Float] temperature The sampling temperature for the model, between 0 and 2.0.
+      #   Higher values make output more random, while lower values make it more focused. Defaults to 0.5.
+      # @param [Hash] additional_options Additional configuration options to pass to the API, such as
+      #   max_tokens, top_p, stop, and others. This allows flexibility for adding any supported parameters
+      #   without modifying the class. The keys should be symbols.
+      #
+      # @example Basic usage with default model and temperature
+      #   config = RSpec::Llama::OpenaiModelConfiguration.new
+      #
+      # @example Passing custom options
+      #   config = RSpec::Llama::OpenaiModelConfiguration.new(
+      #     model: 'gpt-4',
+      #     temperature: 0.8,
+      #     max_tokens: 150,
+      #     stop: ['\n']
+      #   )
+      #
+      # @see https://platform.openai.com/docs/api-reference/chat/create
+      def initialize(model: DEFAULT_MODEL, temperature: DEFAULT_TEMPERATURE, **additional_options)
+        @model = model
+        @temperature = temperature
+        @additional_options = additional_options
+      end
+
+      # Converts the configuration into a hash format for making API requests.
+      #
+      # @return [Hash] A hash representation of the model configuration.
+      #   This hash includes the model, temperature, and any additional options provided during initialization.
+      #   Nil values are omitted from the hash.
+      def to_h
+        { model:, temperature:, **additional_options }.compact
+      end
+    end
+  end
+end

data/lib/rspec/llama/model_runner_results/base_model_runner_result.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    class BaseModelRunnerResult
+      attr_reader :response
+
+      def initialize(response)
+        @response = response
+      end
+
+      def to_s
+        raise NotImplementedError
+      end
+    end
+  end
+end

data/lib/rspec/llama/model_runner_results/llama_cpp_model_runner_result.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    class LlamaCppModelRunnerResult < BaseModelRunnerResult
+      def to_s
+        response.strip
+      end
+    end
+  end
+end

data/lib/rspec/llama/model_runner_results/ollama_model_runner_result.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    class OllamaModelRunnerResult < BaseModelRunnerResult
+      def to_s
+        response
+      end
+    end
+  end
+end

data/lib/rspec/llama/model_runner_results/openai_model_runner_result.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    class OpenaiModelRunnerResult < BaseModelRunnerResult
+      MESSAGE_PATH = [:choices, 0, :message, :content].freeze
+
+      def to_s
+        response.dig(*MESSAGE_PATH)
+      end
+    end
+  end
+end

data/lib/rspec/llama/model_runners/llama_cpp_model_runner.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    class LlamaCppModelRunner
+      DEFAULT_CLI_PATH = 'llama-cli'
+
+      attr_reader :cli_path
+
+      def initialize(cli_path: DEFAULT_CLI_PATH)
+        @cli_path = cli_path
+      end
+
+      # @param [RSpec::Llama::LlamaCppModelConfiguration] configuration
+      # @param [String] prompt
+      # @return [RSpec::Llama::LlamaCppModelRunnerResult]
+      def call(configuration, prompt)
+        command = [cli_path, '--prompt', prompt.to_s] + configuration.to_a
+
+        IO.popen(command, 'r+') do |io|
+          LlamaCppModelRunnerResult.new(io.read)
+        end
+      end
+    end
+  end
+end

data/lib/rspec/llama/model_runners/ollama_model_runner.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    class OllamaModelRunner
+    end
+  end
+end

data/lib/rspec/llama/model_runners/openai_model_runner.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+require 'net/http'
+require 'json'
+
+module RSpec
+  module Llama
+    class OpenaiModelRunner
+      Error = Class.new(StandardError)
+
+      DEFAULT_BASE_URL = 'https://api.openai.com/v1'
+      CHAT_COMPLETIONS_PATH = '/chat/completions'
+
+      # @param [String] access_token
+      # @param [String] base_url
+      # @param [String] organization_id
+      def initialize(access_token:, base_url: DEFAULT_BASE_URL, organization_id: nil, project_id: nil)
+        @access_token = access_token
+        @base_url = base_url
+        @organization_id = organization_id
+        @project_id = project_id
+      end
+
+      # @param [RSpec::Llama::ModelConfiguration] configuration
+      # @param [String] prompt
+      #
+      # @return [RSpec::Llama::OpenaiModelRunnerResult]
+      def call(configuration, prompt)
+        messages = [{ role: 'user', content: prompt.to_s }]
+        response = execute_request(CHAT_COMPLETIONS_PATH, { **configuration.to_h, messages: })
+
+        RSpec::Llama::OpenaiModelRunnerResult.new(response)
+      end
+
+      private
+
+      attr_reader :access_token, :base_url, :organization_id, :project_id
+
+      def request_headers
+        {
+          'Content-Type' => 'application/json',
+          'Authorization' => "Bearer #{access_token}",
+          'OpenAI-Organization' => organization_id,
+          'OpenAI-Project' => project_id
+        }.compact
+      end
+
+      def execute_request(path, params)
+        response = Net::HTTP.post(URI("#{base_url}#{path}"), JSON.dump(params), request_headers)
+        json_response = JSON.parse(response.body, symbolize_names: true)
+        return json_response if response.is_a?(Net::HTTPSuccess)
+
+        raise Error, json_response.dig(:error, :message)
+      end
+    end
+  end
+end

data/lib/rspec/llama/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module RSpec
+  module Llama
+    VERSION = '0.1.0'
+  end
+end

data/lib/rspec/llama.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require 'rspec'
+
+require_relative 'llama/version'
+require_relative 'llama/helpers'
+require_relative 'llama/matchers'
+
+module RSpec
+  module Llama
+    {
+      # Model configurations
+      OpenaiModelConfiguration: 'rspec/llama/model_configurations/openai_model_configuration',
+      LlamaCppModelConfiguration: 'rspec/llama/model_configurations/llama_cpp_model_configuration',
+      OllamaModelConfiguration: 'rspec/llama/model_configurations/ollama_model_configuration',
+
+      # Model runners
+      OpenaiModelRunner: 'rspec/llama/model_runners/openai_model_runner',
+      LlamaCppModelRunner: 'rspec/llama/model_runners/llama_cpp_model_runner',
+      OllamaModelRunner: 'rspec/llama/model_runners/ollama_model_runner',
+
+      # Model runner results
+      BaseModelRunnerResult: 'rspec/llama/model_runner_results/base_model_runner_result',
+      OpenaiModelRunnerResult: 'rspec/llama/model_runner_results/openai_model_runner_result',
+      LlamaCppModelRunnerResult: 'rspec/llama/model_runner_results/llama_cpp_model_runner_result',
+      OllamaModelRunnerResult: 'rspec/llama/model_runner_results/ollama_model_runner_result'
+    }.each { |class_name, path| autoload class_name, path }
+  end
+end
+
+RSpec.configure do |config|
+  config.include RSpec::Llama::Helpers
+  config.include RSpec::Llama::Matchers
+end

data/lib/rspec-llama.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require_relative 'rspec/llama'

metadata

@@ -0,0 +1,90 @@
+--- !ruby/object:Gem::Specification
+name: rspec-llama
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Vadim S.
+- Artur A.
+- Anatoli L.
+- Sergy S.
+bindir: exe
+cert_chain: []
+date: 2024-10-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+description: |
+  RSpec::Llama is a testing framework that allows developers to easily configure, run, and validate
+  AI models such as OpenAI's GPT models, Llama, and others within the RSpec ecosystem.
+
+  With a focus on simplicity and extensibility, RSpec::Llama provides:
+  - A standardized approach to configuring different AI models with customizable parameters.
+  - Runners to execute model interactions and capture responses seamlessly.
+  - Comprehensive assertion types to validate model outputs against expected patterns.
+
+  Whether you are developing AI-powered applications or simply need a reliable way to test various AI
+  models' outputs, RSpec::Llama offers an all-in-one solution that integrates smoothly into your existing RSpec setup.
+email:
+- sergy@cybergizer.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CODE_OF_CONDUCT.md
+- LICENSE
+- README.md
+- lib/rspec-llama.rb
+- lib/rspec/llama.rb
+- lib/rspec/llama/helpers.rb
+- lib/rspec/llama/matchers.rb
+- lib/rspec/llama/matchers/match.rb
+- lib/rspec/llama/matchers/match_all.rb
+- lib/rspec/llama/matchers/match_any.rb
+- lib/rspec/llama/matchers/match_none.rb
+- lib/rspec/llama/model_configurations/llama_cpp_model_configuration.rb
+- lib/rspec/llama/model_configurations/ollama_model_configuration.rb
+- lib/rspec/llama/model_configurations/openai_model_configuration.rb
+- lib/rspec/llama/model_runner_results/base_model_runner_result.rb
+- lib/rspec/llama/model_runner_results/llama_cpp_model_runner_result.rb
+- lib/rspec/llama/model_runner_results/ollama_model_runner_result.rb
+- lib/rspec/llama/model_runner_results/openai_model_runner_result.rb
+- lib/rspec/llama/model_runners/llama_cpp_model_runner.rb
+- lib/rspec/llama/model_runners/ollama_model_runner.rb
+- lib/rspec/llama/model_runners/openai_model_runner.rb
+- lib/rspec/llama/version.rb
+homepage: https://github.com/aifoundry-org/rspec-llama
+licenses:
+- Apache-2.0
+metadata:
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.6.0.dev
+specification_version: 4
+summary: A versatile testing framework for different AI model configurations.
+test_files: []