rails-nl2sql 0.2.6 → 0.2.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 32ceb33965fb596a339bbf54e8945aa3d03df7cd53224076263363694776f699
-  data.tar.gz: ad845d46819e09cf9b2ba74bf24529f4f370829f19a5b1564183ce06b6089188
+  metadata.gz: ef91bcc7db96e148113aa100b915ef59c82d680da31658ebb57ce394b9a48b8d
+  data.tar.gz: 0f8a22ed3c09624dd7464c8de62a3fb98c8aee8fdf5cb6c3fe47b122d5e0bac0
 SHA512:
-  metadata.gz: 00bf7f7132cefed3bfdcf1d3936b2264e495d8db545aac8551d72273454d9d1605e017cecfd1e1c09f6a4a9a7db625502601feb28ac5b5ada18f68e7fe34b584
-  data.tar.gz: efb583c44acee0a05c636f8f462bec47119dbf48e102c9736940e1ecd665ebc90bdde3fb57bf7dba9907e3dbd3e648c752b58196cdbe7af6eea896c85a188329
+  metadata.gz: 83b4a297e39e5caec803253adb5851d55b0b817216acecbd5d79409c4de21705e3e22908ab0224dac504573450c78847477391a0b7c7ccd9c656fa679b6cb8b3
+  data.tar.gz: 0bce6a48bbcf4ce79764f967ab6cd2b1be63af2dc4ae3be3b5184cd194e4b3e7bf85b8e355169df8e3689cef2739ee883c349acce0fbb452379c98fb8f84576f

data/CHANGELOG.md

@@ -7,7 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-
+## [0.2.8] - 2025-07-21
+- Adding support for Google's Gemini LLM provider.
+- Adding a command-line interface (CLI).
+- Improving the documentation.
+- Adding a debug mode.
+- Adding support for the Oracle database adapter.
+- Refactoring the code to make it more modular and easier to understand.
+- Adding more comments to the code.
+ 
+## [0.2.7] - 2025-07-18
+### Fixed
+- Fix from_nl trailing semicolon
 
 ## [0.2.5] - 2025-07-18
 ### Fixed

data/PLAN.md

@@ -0,0 +1,30 @@
+
+# Plan for Improving rails-nl2sql Gem
+
+This document outlines a plan to improve the `rails-nl2sql` gem. The focus is on improving the developer experience, expanding the gem's capabilities, and making it more robust.
+
+## 1. Improve Developer Experience
+
+*   **Add a command-line interface (CLI):** A CLI would make it easier for developers to use the gem without having to write code. The CLI could be used to generate queries, get schema information, and more.
+*   **Improve the documentation:** The documentation should be more comprehensive and include more examples. It should also be easier to navigate.
+*   **Add a debug mode:** A debug mode would make it easier for developers to troubleshoot problems with the gem. The debug mode could log information about the queries that are being generated, the schema that is being used, and more.
+
+## 2. Expand Capabilities
+
+*   **Add support for more database adapters:** The gem currently only supports PostgreSQL, MySQL, and SQLite. It would be beneficial to add support for other popular database adapters, such as Oracle and SQL Server.
+*   **Add support for more LLM providers:** The gem currently only supports OpenAI, Anthropic and Llama. It would be beneficial to add support for other popular LLM providers, such as Google's Gemini.
+*   **Add support for more complex queries:** The gem currently only supports simple SELECT queries. It would be beneficial to add support for more complex queries, such as queries with joins, subqueries, and window functions.
+
+## 3. Make the Gem More Robust
+
+*   **Add more tests:** The gem currently has a good test suite, but it could be improved. More tests should be added to cover all of the gem's functionality.
+*   **Add more error handling:** The gem currently has some error handling, but it could be improved. More error handling should be added to handle all possible errors that could occur.
+*   **Add a performance benchmark:** A performance benchmark would help to ensure that the gem is performing well. The benchmark could be used to track the performance of the gem over time and to identify any performance bottlenecks.
+
+## 4. Improve the Codebase
+
+*   **Refactor the code:** The code could be refactored to make it more modular and easier to understand.
+*   **Add more comments:** The code could be better commented to make it easier for other developers to understand.
+*   **Use a consistent coding style:** The code should be formatted using a consistent coding style.
+
+This is just a high-level plan. More detailed planning will be required for each of these items.

data/README.md

@@ -52,13 +52,24 @@ returns an `ActiveRecord::Relation` so you can chain scopes, pagination and
 other query modifiers as usual.
 
 ```ruby
+# Get all users who signed up last week and limit the results to 10.
 User.from_nl("all users who signed up last week").limit(10)
+
+# Get all users from California and order them by their name.
+User.from_nl("all users from california").order(:name)
+
+# Get all users who have an order with a total greater than 100.
+User.from_nl("all users who have an order with a total greater than 100")
 ```
 
 You can also specify which tables to include or exclude:
 
 ```ruby
+# Get all the orders for the user with email 'test@example.com', but only include the `users` and `orders` tables.
 results = Rails::Nl2sql::Processor.execute("Show me all the orders for the user with email 'test@example.com'", include: ["users", "orders"])
+
+# Get all the orders for the user with email 'test@example.com', but exclude the `payments` table.
+results = Rails::Nl2sql::Processor.execute("Show me all the orders for the user with email 'test@example.com'", exclude: ["payments"])
 ```
 
 ### Getting a list of tables
@@ -96,6 +107,26 @@ Rails::Nl2sql.configure do |config|
 end
 ```
 
+### Llama
+
+You can also use a local Llama-based HTTP endpoint. You'll need to have a local Llama server running.
+
+```ruby
+Rails::Nl2sql.configure do |config|
+  config.provider = Rails::Nl2sql::Providers::LlamaProvider.new(endpoint: "http://localhost:8080/completion")
+end
+```
+
+### Google Gemini
+
+You can also use Google's Gemini models. You'll need to have the `google-apis-aiplatform_v1` gem installed and be authenticated with Google Cloud.
+
+```ruby
+Rails::Nl2sql.configure do |config|
+  config.provider = Rails::Nl2sql::Providers::GeminiProvider.new(api_key: ENV['GOOGLE_API_KEY'])
+end
+```
+
 ## Prompt Templates
 
 The prompts used to talk to the LLM are defined in a YAML/ERB template. You can override this template
@@ -115,12 +146,53 @@ Set the path via `config.prompt_template_path`.
 Large schemas can exceed the model context window. Use `config.max_schema_lines` to automatically truncate
 the schema snippet sent to the model. Only the first N lines are included.
 
+## Debug Mode
+
+This gem provides a debug mode for troubleshooting problems. When debug mode is enabled, the gem will log information about the queries that are being generated, the schema that is being used, and more.
+
+To enable debug mode, set the `debug` option to `true` in your initializer file:
+
+```ruby
+# config/initializers/rails_nl2sql.rb
+Rails::Nl2sql.configure do |config|
+  config.debug = true
+end
+```
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
+## Command-Line Interface (CLI)
+
+This gem provides a command-line interface (CLI) for interacting with the NL2SQL processor.
+
+### `query`
+
+Converts a natural language query to SQL.
+
+```bash
+$ rails-nl2sql query "Show me all the users from California"
+```
+
+### `schema`
+
+Displays the database schema.
+
+```bash
+$ rails-nl2sql schema
+```
+
+### `tables`
+
+Displays the database tables.
+
+```bash
+$ rails-nl2sql tables
+```
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/vancuren/rails-nl2sql.

data/bin/rails-nl2sql

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require_relative '../lib/rails/nl2sql/cli'
+
+Rails::Nl2sql::CLI.start(ARGV)

data/lib/rails/nl2sql/active_record_extension.rb

@@ -9,6 +9,8 @@ module Rails
       class_methods do
         def from_nl(prompt, options = {})
           sql = Rails::Nl2sql::Processor.generate_query_only(prompt, options)
+          sql = sql.to_s.strip
+          sql = sql.chomp(';')
           from(Arel.sql("(#{sql}) AS #{table_name}"))
         end
       end

data/lib/rails/nl2sql/cli.rb

@@ -0,0 +1,23 @@
+require 'thor'
+require 'rails/nl2sql'
+
+module Rails
+  module Nl2sql
+    class CLI < Thor
+      desc "query [NATURAL_LANGUAGE_QUERY]", "Converts a natural language query to SQL"
+      def query(natural_language_query)
+        puts Rails::Nl2sql::Processor.generate_query_only(natural_language_query)
+      end
+
+      desc "schema", "Displays the database schema"
+      def schema
+        puts Rails::Nl2sql::Processor.get_schema
+      end
+
+      desc "tables", "Displays the database tables"
+      def tables
+        puts Rails::Nl2sql::Processor.get_tables
+      end
+    end
+  end
+end

data/lib/rails/nl2sql/prompt_builder.rb

@@ -0,0 +1,29 @@
+require 'erb'
+
+module Rails
+  module Nl2sql
+    class PromptBuilder
+      def self.build(input, db_server, retrieved_context)
+        template = Rails::Nl2sql.prompt_template
+
+        erb_context = Object.new
+        erb_context.instance_variable_set(:@input, input)
+        erb_context.instance_variable_set(:@db_server, db_server)
+        erb_context.instance_variable_set(:@retrieved_context, retrieved_context)
+
+        erb_context.define_singleton_method(:get_binding) do
+          binding
+        end
+
+        erb_context.define_singleton_method(:input) { @input }
+        erb_context.define_singleton_method(:db_server) { @db_server }
+        erb_context.define_singleton_method(:retrieved_context) { @retrieved_context }
+
+        system_prompt = ERB.new(template['system']).result(erb_context.get_binding)
+        user_prompt = ERB.new(template['user']).result(erb_context.get_binding)
+
+        "#{system_prompt}\n\n#{user_prompt}"
+      end
+    end
+  end
+end

data/lib/rails/nl2sql/providers/gemini_provider.rb

@@ -0,0 +1,48 @@
+require 'google/apis/aiplatform_v1'
+
+module Rails
+  module Nl2sql
+    module Providers
+      class GeminiProvider < Base
+        def initialize(api_key:, model: 'gemini-pro')
+          # Authenticate with Google Cloud
+          # You can use a service account or your user credentials.
+          # For more information, see: https://cloud.google.com/docs/authentication
+          Google::Auth.new.apply(scope: 'https://www.googleapis.com/auth/cloud-platform')
+
+          @client = Google::Apis::AiplatformV1::AIPlatformService.new
+          @model = model
+        end
+
+        def complete(prompt:, **params)
+          # The Gemini API uses a different request format than the other providers.
+          # We need to convert the prompt into a format that the Gemini API can understand.
+          request = Google::Apis::AiplatformV1::GoogleCloudAiplatformV1PredictRequest.new(
+            instances: [
+              {
+                prompt: prompt
+              }
+            ],
+            parameters: {
+              temperature: params[:temperature] || 0.2,
+              maxOutputTokens: params[:max_tokens] || 256,
+              topP: params[:top_p] || 0.95,
+              topK: params[:top_k] || 40
+            }
+          )
+
+          # The endpoint for the Gemini API is different for each region.
+          # We need to get the endpoint for the region that the user is in.
+          # For more information, see: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models
+          endpoint = 'us-central1-aiplatform.googleapis.com' # Or another regional endpoint
+
+          response = @client.predict_project_location_publisher_model(endpoint, request)
+
+          # The Gemini API returns a different response format than the other providers.
+          # We need to convert the response into a format that the other providers can understand.
+          response.predictions.first['content']
+        end
+      end
+    end
+  end
+end

data/lib/rails/nl2sql/query_generator.rb

@@ -1,20 +1,32 @@
 require "erb"
 require "yaml"
+require "rails/nl2sql/prompt_builder"
+
 module Rails
   module Nl2sql
     class QueryGenerator
       DEFAULT_MODEL = 'gpt-3.5-turbo-instruct'
 
+      # Initializes a new QueryGenerator.
+      #
+      # @param provider [Object] The AI provider to use for generating queries.
+      # @param model [String] The name of the AI model to use.
       def initialize(provider: nil, model: DEFAULT_MODEL)
         @provider = provider || Rails::Nl2sql.provider || default_provider(model)
         @model = model
       end
 
+      # Generates a SQL query from a natural language prompt.
+      #
+      # @param prompt [String] The natural language prompt.
+      # @param schema [String] The database schema.
+      # @param db_server [String] The type of database server.
+      # @param tables [Array<String>] The tables to include in the schema.
+      # @return [String] The generated SQL query.
       def generate_query(prompt, schema, db_server = 'PostgreSQL', tables = nil)
         retrieved_context = build_context(schema, tables)
 
-        system_prompt, user_prompt = build_prompts(prompt, db_server, retrieved_context)
-        full_prompt = "#{system_prompt}\n\n#{user_prompt}"
+        full_prompt = PromptBuilder.build(prompt, db_server, retrieved_context)
 
         response = @provider.complete(prompt: full_prompt, max_tokens: 500, temperature: 0.1)
         generated_query = extract_text(response)
@@ -27,10 +39,18 @@ module Rails
 
       private
 
+      # Returns the default AI provider.
+      #
+      # @param model [String] The name of the AI model to use.
+      # @return [Object] The default AI provider.
       def default_provider(model)
         Providers::OpenaiProvider.new(api_key: Rails::Nl2sql.api_key, model: model)
       end
 
+      # Extracts the text from the AI provider's response.
+      #
+      # @param response [Object] The response from the AI provider.
+      # @return [String] The extracted text.
       def extract_text(response)
         if response.is_a?(Hash)
           response.dig('choices', 0, 'text')&.strip
@@ -39,6 +59,11 @@ module Rails
         end
       end
 
+      # Builds the context for the AI prompt.
+      #
+      # @param schema [String] The database schema.
+      # @param tables [Array<String>] The tables to include in the schema.
+      # @return [String] The context for the AI prompt.
       def build_context(schema, tables)
         context = if tables&.any?
           filter_schema_by_tables(schema, tables)
@@ -48,6 +73,10 @@ module Rails
         apply_context_window(context)
       end
 
+      # Applies the context window to the context.
+      #
+      # @param context [String] The context for the AI prompt.
+      # @return [String] The context with the context window applied.
       def apply_context_window(context)
         max_lines = Rails::Nl2sql.max_schema_lines
         return context unless max_lines
@@ -58,6 +87,11 @@ module Rails
         lines.first(max_lines).join("\n")
       end
 
+      # Filters the schema by the given tables.
+      #
+      # @param schema [String] The database schema.
+      # @param tables [Array<String>] The tables to include in the schema.
+      # @return [String] The filtered schema.
       def filter_schema_by_tables(schema, tables)
         lines = schema.split("\n")
         filtered_lines = []
@@ -76,36 +110,17 @@ module Rails
         filtered_lines.join("\n")
       end
 
-      def build_prompts(input, db_server, retrieved_context)
-        template = Rails::Nl2sql.prompt_template
-        
-        # Create ERB context with explicit local variables
-        erb_context = Object.new
-        erb_context.instance_variable_set(:@input, input)
-        erb_context.instance_variable_set(:@db_server, db_server)
-        erb_context.instance_variable_set(:@retrieved_context, retrieved_context)
-        
-        erb_context.define_singleton_method(:get_binding) do
-          binding
-        end
-        
-        # Define accessor methods for the ERB template
-        erb_context.define_singleton_method(:input) { @input }
-        erb_context.define_singleton_method(:db_server) { @db_server }
-        erb_context.define_singleton_method(:retrieved_context) { @retrieved_context }
-        
-        system_prompt = ERB.new(template['system']).result(erb_context.get_binding)
-        user_prompt = ERB.new(template['user']).result(erb_context.get_binding)
-        [system_prompt, user_prompt]
-      end
-
+      # Cleans the SQL response from the AI provider.
+      #
+      # @param query [String] The SQL query to clean.
+      # @return [String] The cleaned SQL query.
       def clean_sql_response(query)
         return query unless query
 
         query = query.gsub(/```sql\n?/, '')
         query = query.gsub(/```\n?/, '')
         query = query.strip
-        query = query.gsub(/^.*?(SELECT|WITH|INSERT|UPDATE|DELETE|CREATE|DROP|ALTER)/i, '\1')
+        query = query.gsub(/^(.*?)(SELECT|WITH|INSERT|UPDATE|DELETE|CREATE|DROP|ALTER)/i, '\1')
         lines = query.split("\n")
         sql_lines = []
 
@@ -122,6 +137,9 @@ module Rails
         cleaned_query
       end
 
+      # Validates the safety of the SQL query.
+      #
+      # @param query [String] The SQL query to validate.
       def validate_query_safety(query)
         return unless query
 

data/lib/rails/nl2sql/schema_builder.rb

@@ -30,7 +30,7 @@ module Rails
           'MySQL'
         when 'sqlite3'
           'SQLite'
-        when 'oracle'
+        when 'oracle', 'oracleenhanced'
           'Oracle'
         when 'sqlserver'
           'SQL Server'

data/lib/rails/nl2sql/version.rb

@@ -1,5 +1,5 @@
 module Rails
   module Nl2sql
-    VERSION = "0.2.6"
+    VERSION = "0.2.8"
   end
 end

data/lib/rails/nl2sql.rb

@@ -3,6 +3,7 @@ require "rails/nl2sql/providers/base"
 require "rails/nl2sql/providers/openai_provider"
 require "rails/nl2sql/providers/anthropic_provider"
 require "rails/nl2sql/providers/llama_provider"
+require "rails/nl2sql/providers/gemini_provider"
 require "rails/nl2sql/query_generator"
 require "rails/nl2sql/schema_builder"
 require "rails/nl2sql/query_validator"
@@ -20,6 +21,7 @@ module Rails
       attr_accessor :model
       attr_accessor :provider
       attr_accessor :max_schema_lines
+      attr_accessor :debug
 
       def prompt_template_path=(path)
         @prompt_template = nil
@@ -33,6 +35,7 @@ module Rails
 
     @model = 'gpt-3.5-turbo-instruct'
     @max_schema_lines = 200
+    @debug = false
 
     def self.configure
       yield self
@@ -46,19 +49,7 @@ module Rails
 
     class Processor
       def self.execute(natural_language_query, options = {})
-        db_server = SchemaBuilder.get_database_type
-        schema = SchemaBuilder.build_schema(options)
-        tables = options[:tables]
-
-        query_generator = QueryGenerator.new(model: Rails::Nl2sql.model)
-        generated_query = query_generator.generate_query(
-          natural_language_query,
-          schema,
-          db_server,
-          tables
-        )
-
-        QueryValidator.validate(generated_query)
+        generated_query = self.generate_query_only(natural_language_query, options)
         ActiveRecord::Base.connection.execute(generated_query)
       end
 
@@ -67,6 +58,13 @@ module Rails
         schema = SchemaBuilder.build_schema(options)
         tables = options[:tables]
 
+        if Rails::Nl2sql.debug && defined?(Rails)
+          Rails.logger.debug "--- Rails NL2SQL Debug Info ---"
+          Rails.logger.debug "Natural Language Query: #{natural_language_query}"
+          Rails.logger.debug "Schema: \n#{schema}"
+          Rails.logger.debug "Tables: #{tables.inspect}" if tables
+        end
+
         query_generator = QueryGenerator.new(model: Rails::Nl2sql.model)
         generated_query = query_generator.generate_query(
           natural_language_query,
@@ -75,6 +73,11 @@ module Rails
           tables
         )
 
+        if Rails::Nl2sql.debug && defined?(Rails)
+          Rails.logger.debug "Generated Query: #{generated_query}"
+          Rails.logger.debug "--- End Rails NL2SQL Debug Info ---"
+        end
+
         QueryValidator.validate(generated_query)
         generated_query
       end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: rails-nl2sql
 version: !ruby/object:Gem::Version
-  version: 0.2.6
+  version: 0.2.8
 platform: ruby
 authors:
 - Russell Van Curen
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2025-07-18 00:00:00.000000000 Z
+date: 2025-07-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: openai
@@ -123,9 +123,11 @@ files:
 - Gemfile
 - Gemfile.lock
 - LICENSE.txt
+- PLAN.md
 - README.md
 - Rakefile
 - bin/console
+- bin/rails-nl2sql
 - bin/setup
 - lib/generators/rails/nl2sql/install_generator.rb
 - lib/generators/rails/nl2sql/templates/rails_nl2sql.rb
@@ -134,9 +136,12 @@ files:
 - lib/rails-nl2sql.rb
 - lib/rails/nl2sql.rb
 - lib/rails/nl2sql/active_record_extension.rb
+- lib/rails/nl2sql/cli.rb
+- lib/rails/nl2sql/prompt_builder.rb
 - lib/rails/nl2sql/prompts/default.yml.erb
 - lib/rails/nl2sql/providers/anthropic_provider.rb
 - lib/rails/nl2sql/providers/base.rb
+- lib/rails/nl2sql/providers/gemini_provider.rb
 - lib/rails/nl2sql/providers/llama_provider.rb
 - lib/rails/nl2sql/providers/openai_provider.rb
 - lib/rails/nl2sql/query_generator.rb