LLMTape 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 070ff888788202111d91ae3eff13c210d3c54fbcaa75f48f5aeb6638006f3d5a
+  data.tar.gz: 2448eb31c0514b95e3b8ee56a648ac956dd4d69b11c98eb2f5ebeea8cd1a4b33
+SHA512:
+  metadata.gz: 4416d4459d88b1b7570992949caa2cc77358bcc44412f75bcd00d98f5842b88e973d4c0b9678ccfb91480c760fbb9bf3aa6ae710719473460c8afd5dfacfe476
+  data.tar.gz: 68ec916b33e31c130ea59b857db371c2bce6324e82940f12b583546091174afb7119b720a028f22d70419427433512b9a084be084da7bf41781935338b8638f8

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Amit Leshed
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,101 @@
+<p align="center">
+  <img src="logo_white.png" alt="LLMTape Logo" width="250"/>
+</p>
+
+# 🎥 LLMTape
+
+**LLMTape** is a lightweight Ruby gem for testing code that calls Large Language Models.
+It works like a cassette tape: record an API call once, then replay it forever.
+
+It wraps any LLM client with a tiny DSL. In test environement, it records “tapes” (YAML fixtures of real LLM calls) and replays them on subsequent runs; when a tape is stale, it re-records to keep tests current. Production stays clean and safe, while CI avoids hammering the API every run--yielding deterministic tests, faster pipelines, and fewer tokens spent.
+
+---
+
+## 🌟 Features
+
+**🎥 Record & Replay** – Capture any LLM API call once, then replay it in future test runs.
+
+**🔒 Deterministic Tests** – Eliminate flaky outputs and unstable CI runs caused by live LLM calls.
+
+**📼 Human-Readable Fixtures** – Responses are stored as YAML “tapes” you can inspect, diff, and commit.
+
+**⚡ Fast & Cheap** – No repeated API calls in CI; only re-record when your prompt or logic changes.
+
+**🧪 Client-Agnostic** – Works with any Ruby LLM client (e.g. ruby_llm), since you wrap the call yourself.
+
+**⏳ Staleness Detection** – Marks a tape as stale if it’s too old or if the prompt has changed.
+
+**🔧 Simple DSL** – Just wrap your LLM call in LLMTape.use("my_call") { ... } and it handles the rest.
+
+**🚀 CI-Friendly** – Run in replay mode to avoid network calls and API keys entirely on CI.
+
+---
+
+## 📦 Installation
+
+Add this gem to your application's `Gemfile`:
+
+```bash
+gem "LLMTape", "~> 0.1.0" # in your gemfile
+bundle add LLMTape        # or through cli
+bundle install
+```
+
+## ⚙️ Configuration
+```ruby
+require "llm_tape"
+
+LLMTape.configure(
+  fixtures_directory_path: "tmp/fixtures/llm",
+  mode: (ENV["LLM_TAPE"] || "auto").to_sym # :record, :replay, or :auto
+)
+```
+
+## 📖 Usage
+```ruby
+class MyFancyAIController < ApplicationController
+  def create
+    prompt = params[:prompt].presence || "LLM Prompt"
+    LLMTape.use("my_fancy_ai#create", request: { prompt: prompt }) do
+      RubyLLM.chat.ask(prompt) # Say you use RubyLLM (it's great you should)
+    end
+  end
+
+  # LLMTape records/replays only in test; it enforces the env check internally.
+  # Dev/Prod behavior is unchanged (straight passthrough). No extra guards needed.
+end
+
+class LLMTest < Minitest::Test
+  def test_three_word_greeting
+    prompt = "Say hello in exactly three words."
+    result = LLMTape.use("three_word_greeting", request: { prompt: prompt }) do
+      RubyLLM.chat.ask(prompt)
+    end
+
+    refute_empty result.to_s.strip
+  end
+end
+```
+
+## 🖥️ CLI
+```bash
+LLM_TAPE=record/replay/auto bundle exec rake test
+```
+
+## 🛠 Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## 🤝 Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/LLMTape. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/LLMTape/blob/main/CODE_OF_CONDUCT.md).
+
+## 📜 License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## 🧭 Code of Conduct
+
+Everyone interacting in the LLMTape project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/LLMTape/blob/main/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "minitest/test_task"
+
+Minitest::TestTask.create
+
+task default: :test

data/lib/LLMTape/concerns/record_initializer.rb

@@ -0,0 +1,31 @@
+module LLMTape
+  module Backpack
+    module RecordInitializer
+      def self.included(base)
+        base.extend(ClassMethods)
+      end
+
+      module ClassMethods
+        def call(**kwargs)
+          new(**defaults.merge(kwargs)).call
+        end
+
+        private
+
+        def defaults
+          { metadata: {}, path: LLMTape.fixtures_directory_path }
+        end
+      end
+
+      def initialize(description:, request:, response:, metadata: {}, path: LLMTape.fixtures_directory_path)
+        @description = description
+        @request     = request
+        @response    = response
+        @metadata    = metadata
+        @path        = path
+      end
+
+      attr_reader :description, :request, :response, :metadata, :path
+    end
+  end
+end

data/lib/LLMTape/concerns/replay_initializer.rb

@@ -0,0 +1,29 @@
+module LLMTape
+  module Backpack
+    module ReplayInitializer
+      def self.included(base)
+        base.extend(ClassMethods)
+      end
+
+      module ClassMethods
+        def call(**kwargs)
+          new(**defaults.merge(kwargs)).call
+        end
+
+        private
+
+        def defaults
+          { path: LLMTape.fixtures_directory_path }
+        end
+      end
+
+      def initialize(description:, request:, path: File.join(LLMTape.fixtures_directory_path, "llm_tapes.yml"))
+        @description  = description
+        @request      = request
+        @fixture_path = path
+      end
+
+      attr_reader :description, :request, :fixture_path
+    end
+  end
+end

data/lib/LLMTape/services/record.rb

@@ -0,0 +1,65 @@
+require "digest"
+require "fileutils"
+require "yaml"
+require "time"
+require_relative "../concerns/record_initializer"
+
+module LLMTape
+  module Services
+    class Record
+      attr_reader :description, :request, :response, :metadata
+
+      include LLMTape::Backpack::RecordInitializer
+
+      def call
+        generate_fixture
+      end
+
+      private
+      
+      def generate_fixture
+        existing_created_at = @metadata["created_at"] || @metadata[:created_at]
+        fixture_data        = build_fixture_data(@description, @request, @response, @metadata, existing_created_at)
+        file_exists         = File.exist?(file_path)
+        fixtures            = file_exists ? YAML.load_stream(File.read(file_path)) : []
+        matched_fixture     = fixtures.find_index { |f| f["description"] == @description }
+
+        fixtures[matched_fixture] = fixture_data if matched_fixture
+        fixtures << fixture_data                 unless matched_fixture
+      
+        record_dat_tape(file_path, fixtures, fixture_data)
+      end
+
+      def build_fixture_data(description, request, response, metadata, existing_created_at = nil)
+        {
+          "description" => description,
+          "data"        => {
+            "id"       => generate_hash(description),
+            "request"  => request,
+            "response" => response,
+            "metadata" => metadata.merge({ :created_at => existing_created_at || Time.now.utc.iso8601 })
+          }
+        }
+      end
+
+      def record_dat_tape(file_path, fixtures, fixture_data)
+        File.open(file_path, "w") do |file|
+          fixtures.each { |fixture| file.write(fixture.to_yaml) }
+        end
+
+        fixture_data
+      end
+
+      def generate_hash(description)
+        description.downcase.split.join("_") + "_" + Digest::SHA256.hexdigest(description)[0..7]
+      end
+
+      def file_path
+        fixture_file_path = File.join(
+          LLMTape.fixtures_directory_path,
+          "llm_tapes.yml"
+        )
+      end
+    end
+  end
+end

data/lib/LLMTape/services/replay.rb

@@ -0,0 +1,16 @@
+require_relative "../concerns/replay_initializer"
+
+module LLMTape
+  module Services
+    class Replay
+      attr_reader :description, :request
+
+      include LLMTape::Backpack::ReplayInitializer
+
+      def call
+        LLMTape::Backpack.find_tape(@fixture_path, @description) ||
+        raise(ArgumentError, "Tape not found for description: #{@description}")
+      end
+    end
+  end
+end

data/lib/LLMTape/services/stale_buster.rb

@@ -0,0 +1,44 @@
+require "yaml"
+require "debug"
+
+module LLMTape
+  module Services
+    class StaleBuster
+      def self.call(description, current_prompt)
+        @current_prompt = current_prompt
+        @description    = description
+
+        stale?(current_prompt: @current_prompt)
+      end
+
+      private
+
+      def self.stale?(current_prompt:)
+        return true unless file_exists?
+        fixture      = LLMTape::Backpack.find_tape(fixture_path, @description)
+        return true unless fixture
+        created_at   = fixture.dig("data", "metadata", :created_at)&.to_s
+        return true unless created_at
+        created_time = Time.parse(created_at) rescue nil
+        return true unless created_time
+      
+        too_old      = (Time.now - created_time) > 30*24*60*60
+        prompt_diff  = fixture.dig("data", "request", :prompt).to_s != current_prompt.to_s
+      
+        too_old || prompt_diff
+      end
+      
+
+      def self.fixture_path
+        File.join(
+          LLMTape.fixtures_directory_path,
+          "llm_tapes.yml"
+        )
+      end
+
+      def self.file_exists?
+        File.exist?(fixture_path)
+      end
+    end
+  end
+end

data/lib/LLMTape/services/utilities.rb

@@ -0,0 +1,34 @@
+require "yaml"
+require "digest"
+
+module LLMTape
+  module Backpack
+    def self.generate_hash(description)
+      require "digest"
+      Digest::SHA256.hexdigest(description)[0..7]
+    end
+
+    def self.create_file
+      fixture_file_path = File.join(
+        LLMTape.fixtures_directory_path,
+        "llm_tapes.yml"
+      )
+
+      FileUtils.mkdir_p(LLMTape.fixtures_directory_path) unless Dir.exist?(LLMTape.fixtures_directory_path)
+      FileUtils.touch(fixture_file_path) unless File.exist?(fixture_file_path)
+      fixture_file_path
+    end
+
+    def self.find_tape(fixture_path, description)
+      return nil unless File.exist?(fixture_path)
+
+      fixture_path = File.join(LLMTape.fixtures_directory_path, "llm_tapes.yml")
+      create_file unless File.exist?(fixture_path)
+      docs = YAML.load_stream(File.read(fixture_path)).compact
+      docs = [YAML.safe_load(File.read(fixture_path))].compact if docs.empty?
+      target = description.to_s
+
+      docs.find { |doc| (doc["description"] || doc[:description]).to_s == target }
+    end
+  end
+end

data/lib/LLMTape/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module LLMTape
+  VERSION = "0.3.0"
+end

data/lib/LLMTape.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+require_relative "LLMTape/version"
+require_relative "LLMTape/services/record"
+require_relative "LLMTape/services/replay"
+require_relative "LLMTape/services/stale_buster"
+require_relative "LLMTape/services/utilities"
+
+module LLMTape
+  Record      = Services::Record
+  Replay      = Services::Replay
+  StaleBuster = Services::StaleBuster
+
+  DEFAULT_FIXTURES_PATH = "test/fixtures/llm"
+  DEFAULT_MODE          = (ENV["LLMTape"] || "auto").to_sym
+
+  class << self
+    attr_accessor :fixtures_directory_path, :mode
+    
+    def use(description, record: false, request: nil, &block)
+      safety_first!(description, &block)
+      return block.call unless test_env?
+
+      configure if fixtures_directory_path.to_s.empty?
+      setup(description, record, request, &block)
+      use_tape(description)
+    end
+
+    def configure(fixtures_directory_path: DEFAULT_FIXTURES_PATH, mode: DEFAULT_MODE)
+      configure_fixtures_directory(fixtures_directory_path:, mode:)
+    end
+
+    private
+
+    def test_env?
+      ENV["RAILS_ENV"] == "test" || ENV["RACK_ENV"] == "test" || ENV["ENV"] == "test"
+    end
+
+    def safety_first!(description, &block)
+      raise ArgumentError, "You must provide a block" unless block_given?
+      raise ArgumentError, "Description is required" if description.to_s.strip.empty?
+    end
+
+    def setup(description, record, request, &block)
+      @fixture_path     = File.join(fixtures_directory_path, "#{description}.yml")
+      @current_request  = request || {}
+      @current_response = block.call
+      @operation_mode   = record ? :record : mode
+      @stale            = StaleBuster.call(description, @current_request[:prompt])
+    end
+
+    def configure_fixtures_directory(fixtures_directory_path: DEFAULT_FIXTURES_PATH, mode: DEFAULT_MODE)
+      self.fixtures_directory_path = fixtures_directory_path.to_s
+      FileUtils.mkdir_p(self.fixtures_directory_path)
+      self.mode = mode
+    end
+
+    def use_tape(description)
+      if should_record?
+        Record.call(
+          description: description,
+          request:     @current_request,
+          response:    @current_response,
+          metadata:    { fixture_path: @fixture_path, mode: @operation_mode }
+        )
+      elsif should_replay?
+        Replay.call(
+          description: description,
+          request:     @current_request
+        )
+      end
+    end
+
+    def should_replay?
+      (@operation_mode == :replay || @operation_mode == :auto) && !@stale
+    end
+
+    def should_record?
+      @operation_mode == :record || @stale
+    end
+  end
+end

data/sig/LLMTape.rbs

@@ -0,0 +1,4 @@
+module LLMTape
+  VERSION: String
+  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+end

metadata

@@ -0,0 +1,69 @@
+--- !ruby/object:Gem::Specification
+name: LLMTape
+version: !ruby/object:Gem::Version
+  version: 0.3.0
+platform: ruby
+authors:
+- Amit Leshed
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2025-09-01 00:00:00.000000000 Z
+dependencies: []
+description: It wraps any LLM client with a tiny DSL. In test environement, it records
+  “tapes” (YAML fixtures of real LLM calls) and replays them on subsequent runs; when
+  a tape is stale, it re-records to keep tests current. Production stays clean and
+  safe, while CI avoids hammering the API every run--yielding deterministic tests,
+  faster pipelines, and fewer tokens spent.
+email:
+- amitleshed@icloud.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".env"
+- CODE_OF_CONDUCT.md
+- LICENSE.txt
+- LLMTape-0.1.0.gem
+- LLMTape-0.2.0.gem
+- README.md
+- Rakefile
+- lib/LLMTape.rb
+- lib/LLMTape/concerns/record_initializer.rb
+- lib/LLMTape/concerns/replay_initializer.rb
+- lib/LLMTape/services/record.rb
+- lib/LLMTape/services/replay.rb
+- lib/LLMTape/services/stale_buster.rb
+- lib/LLMTape/services/utilities.rb
+- lib/LLMTape/version.rb
+- logo.png
+- logo_white.png
+- sig/LLMTape.rbs
+homepage: https://www.amitleshed.com
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+  homepage_uri: https://www.amitleshed.com
+  source_code_uri: https://www.amitleshed.com
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.4.1
+signing_key:
+specification_version: 4
+summary: 'LLMTape is a lightweight Ruby gem for testing code that calls Large Language
+  Models. It works like a cassette tape: record an API call once, then replay it forever.'
+test_files: []