rails_mcp_code_search 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 6e1c670e94a061746f627ec212078736dca6e7398a6ca1260872eb829ee53d31
+  data.tar.gz: 38cfa6d2f88765590d1db57432e5639637c0f4ef3a4fa075f9e668d54a423bb4
+SHA512:
+  metadata.gz: 4e06c7f98f2bdaca1c3169d48d9753fdd0456ca3cc3e42f1e7951afd36c88dd704e2e00efe2b2ac268650dce37509c4488a9ee5e9039d9a0562e430dae01e845
+  data.tar.gz: 20a0b55ad4f2a8e1498697d3b16941a3361c1165b1e17a6eb1ef15aca82d5cf85f777de415810bf042c847733b7553f2df5911bcd7c6948acff94520db48c942

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Daniel Lopez Prat
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,176 @@
+<div align="center">
+
+  <h1 style="margin-top: 10px;">rails_mcp_code_search</h1>
+
+  <h3>Semantic codebase search for Claude Code via MCP</h3>
+
+  <p>Think Cursor's codebase indexing, but for Claude Code.</p>
+
+  <div align="center">
+    <a href="https://github.com/6temes/rails_mcp_code_search/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/badge/license-MIT-green"/></a>
+    <a href="https://rubygems.org/gems/rails_mcp_code_search"><img alt="Gem Version" src="https://img.shields.io/gem/v/rails_mcp_code_search"/></a>
+    <a href="https://github.com/6temes/rails_mcp_code_search/actions"><img alt="CI" src="https://github.com/6temes/rails_mcp_code_search/actions/workflows/ci.yml/badge.svg"/></a>
+  </div>
+
+  <p>
+    <a href="#how-it-works">How It Works</a> ◆
+    <a href="#quick-start">Quick Start</a> ◆
+    <a href="#tools">Tools</a> ◆
+    <a href="#configuration">Configuration</a> ◆
+    <a href="#architecture">Architecture</a>
+  </p>
+
+</div>
+
+---
+
+## How It Works
+
+- **Ruby files** — Parsed with [Prism](https://ruby.github.io/prism/) into classes, modules, and methods
+- **ERB templates** — Parsed with [Herb](https://github.com/marcoroth/herb) into blocks, conditionals, and HTML elements
+- **Other files** (JS, TS, YAML, Markdown) — Sliding window chunking
+- **Embeddings** — Generated locally with [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) (384 dimensions, zero config)
+- **Vector search** — SQLite + [sqlite-vec](https://alexgarcia.xyz/sqlite-vec/) for cosine similarity
+- **Background indexing** — Starts automatically, search is available immediately
+
+## Quick Start
+
+```sh
+gem install rails_mcp_code_search
+```
+
+Requires Ruby 4.0+. The first run downloads the embedding model (~80 MB) to `~/.cache/informers/`.
+
+Add to `~/.claude/settings.json`:
+
+```json
+{
+  "mcpServers": {
+    "code-search": {
+      "command": "rails-mcp-code-search"
+    }
+  }
+}
+```
+
+That's it. Claude Code launches the server automatically when you open a project.
+
+## Tools
+
+### search
+
+Search the codebase by concept or behavior using natural language.
+
+```text
+query: "user authentication logic"
+limit: 10
+file_pattern: "app/models/**/*.rb"
+```
+
+Returns ranked results with file path, line range, similarity score, and code content. On each search, changed files are automatically re-indexed with a 200ms time budget.
+
+### reindex
+
+Trigger a manual reindex. Returns immediately — runs in the background.
+
+```text
+full: true    # Rebuild entire index
+full: false   # Incremental (default, only changed files)
+```
+
+### status
+
+Check index health, chunk count, embedding provider, and search stats.
+
+## Configuration
+
+### Environment Variables
+
+| Variable | Default | Description |
+|---|---|---|
+| `RAILS_MCP_CODE_SEARCH_PROVIDER` | `local` | Embedding provider: `local` or `openai` |
+| `RAILS_MCP_CODE_SEARCH_DB_PATH` | auto | Override database file path |
+| `RAILS_MCP_CODE_SEARCH_LOG_LEVEL` | `info` | Log level: `debug`, `info`, `warn`, `error` |
+| `RAILS_MCP_CODE_SEARCH_OPENAI_API_KEY` | — | Required when provider is `openai` |
+
+### OpenAI Provider
+
+For faster indexing or higher-dimensional embeddings, use OpenAI's `text-embedding-3-small` (1536 dimensions):
+
+```sh
+export RAILS_MCP_CODE_SEARCH_PROVIDER=openai
+export RAILS_MCP_CODE_SEARCH_OPENAI_API_KEY=sk-...
+```
+
+| | Local (default) | OpenAI |
+|---|---|---|
+| Dimensions | 384 | 1536 |
+| Speed | ~5 min / 1000 files | ~15s / 1000 files |
+| Cost | Free | Per-token API cost |
+| Privacy | Everything stays local | Code sent to OpenAI |
+| Setup | Zero config | Requires API key |
+
+Switching providers triggers a full reindex automatically.
+
+> **Privacy notice:** When using the OpenAI provider, source code chunks from your repository are sent to OpenAI's embedding API. The local provider (default) keeps everything on your machine.
+
+### Supported File Types
+
+`*.rb` `*.erb` `*.js` `*.ts` `*.yml` `*.yaml` `*.md`
+
+Excluded: `vendor/` `node_modules/` `tmp/` `log/`
+
+## Architecture
+
+```text
+┌─────────────────────────────────────────────────────────┐
+│                    Claude Code                          │
+│                  (MCP Client)                           │
+└──────────────────────┬──────────────────────────────────┘
+                       │ stdio
+                       ▼
+┌─────────────────────────────────────────────────────────┐
+│  MCP Server (search, reindex, status)                   │
+└──────────────────────┬──────────────────────────────────┘
+                       │
+          ┌────────────┴────────────┐
+          ▼                         ▼
+┌──────────────────┐     ┌──────────────────┐
+│  Background       │     │  Embedding        │
+│  Worker           │     │  Adapter          │
+│  (sole writer)    │     │  (local / openai) │
+└────────┬─────────┘     └────────┬─────────┘
+         │                        │
+         ▼                        ▼
+┌─────────────────────────────────────────────┐
+│  SQLite + sqlite-vec                        │
+│  (WAL mode, per-project DB)                 │
+│  ~/.local/share/rails-mcp-code-search/      │
+└─────────────────────────────────────────────┘
+```
+
+### Parsers
+
+| File Type | Parser | Chunk Types |
+|-----------|--------|-------------|
+| `*.rb` | Prism AST | `class`, `module`, `method` |
+| `*.erb` | Herb AST | `erb_block`, `erb_conditional`, `html_element` |
+| Everything else | Sliding window | `window` (50 lines, 10 overlap) |
+
+### Key Design Decisions
+
+- **Standalone ActiveRecord** — No Rails runtime dependency, just SQLite
+- **Single writer thread** — All DB mutations go through the background worker
+- **Smart reindex** — Changed files (via `git diff`) are re-indexed before each search
+- **Per-project database** — SHA256 of the project path, stored in `~/.local/share/`
+- **Prism + Herb AST** — Semantic chunking produces better search results than naive line splitting
+
+## License
+
+MIT
+
+---
+
+<div align="center">
+  <p>Made in Tokyo with ❤️ and 🤖</p>
+</div>

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task default: :test

data/exe/rails-mcp-code-search

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require "rails_mcp_code_search"
+
+RailsMcpCodeSearch::Server.start

data/lib/rails_mcp_code_search/background_worker.rb

@@ -0,0 +1,150 @@
+module RailsMcpCodeSearch
+  class BackgroundWorker
+    REINDEX_COOLDOWN = 60
+
+    attr_reader :state, :progress, :errors
+
+    def initialize(indexer:, logger: nil)
+      @indexer = indexer
+      @logger = logger
+      @queue = Queue.new
+      @mutex = Mutex.new
+      @state = :idle
+      @progress = 0.0
+      @errors = []
+      @stop_requested = false
+      @last_full_reindex_at = nil
+      @hit_count_buffer = []
+      @search_count = 0
+      @reindex_count = 0
+    end
+
+    def start
+      @thread = Thread.new { run }
+      enqueue(:full_index)
+      self
+    end
+
+    def stop
+      @stop_requested = true
+      @queue.push(:shutdown)
+      @thread&.join(5)
+    end
+
+    def enqueue(work_type, payload: nil)
+      @queue.push({ type: work_type, payload: })
+    end
+
+    def enqueue_hit_counts(chunk_ids)
+      @mutex.synchronize { @hit_count_buffer.concat(chunk_ids) }
+    end
+
+    def increment_search_count
+      @mutex.synchronize { @search_count += 1 }
+    end
+
+    def increment_reindex_count
+      @mutex.synchronize { @reindex_count += 1 }
+    end
+
+    def wait_for_reindex(timeout: 0.2)
+      deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout
+      loop do
+        return true if @mutex.synchronize { @state } == :idle
+        remaining = deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        return false if remaining <= 0
+        sleep([ remaining, 0.05 ].min)
+      end
+    end
+
+    private
+
+    def run
+      ActiveRecord::Base.connection_pool.with_connection do
+        loop do
+          break if @stop_requested
+
+          work = begin
+            @queue.pop(true)
+          rescue ThreadError
+            flush_counters
+            sleep 0.1
+            next
+          end
+
+          break if work == :shutdown
+
+          process_work(work)
+        end
+      end
+    rescue => e
+      set_state :error
+      @mutex.synchronize { @errors << e.message }
+      log(:error, "Background worker crashed: #{e.message}")
+    end
+
+    def process_work(work)
+      case work[:type]
+      when :full_index
+        return if on_cooldown?
+        set_state :indexing
+        @indexer.index_all
+        @mutex.synchronize do
+          @errors = @indexer.errors
+          @last_full_reindex_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          @reindex_count += 1
+        end
+        set_state :idle
+      when :index_files
+        set_state :indexing
+        @indexer.index_files(work[:payload])
+        @mutex.synchronize { @errors = @indexer.errors }
+        set_state :idle
+      when :flush
+        flush_counters
+      end
+    end
+
+    def on_cooldown?
+      @mutex.synchronize do
+        return false unless @last_full_reindex_at
+        elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - @last_full_reindex_at
+        elapsed < REINDEX_COOLDOWN
+      end
+    end
+
+    def flush_counters
+      ids = @mutex.synchronize { @hit_count_buffer.shift(@hit_count_buffer.size) }
+      if ids.any?
+        Chunk.where(id: ids).update_all("hit_count = hit_count + 1")
+      end
+
+      search_count, reindex_count = @mutex.synchronize do
+        counts = [ @search_count, @reindex_count ]
+        @search_count = 0
+        @reindex_count = 0
+        counts
+      end
+
+      if search_count > 0
+        current = Database::Metadata.get("total_searches").to_i
+        Database::Metadata.set "total_searches", current + search_count
+        Database::Metadata.set "last_search_at", Time.now.iso8601
+      end
+
+      if reindex_count > 0
+        current = Database::Metadata.get("total_reindexes").to_i
+        Database::Metadata.set "total_reindexes", current + reindex_count
+        Database::Metadata.set "last_reindex_at", Time.now.iso8601
+      end
+    end
+
+    def set_state(new_state)
+      @mutex.synchronize { @state = new_state }
+    end
+
+    def log(level, message)
+      @logger&.send(level, message)
+    end
+  end
+end

data/lib/rails_mcp_code_search/chunk.rb

@@ -0,0 +1,7 @@
+module RailsMcpCodeSearch
+  class Chunk < ActiveRecord::Base
+    self.table_name = "chunks"
+
+    has_neighbors :embedding
+  end
+end

data/lib/rails_mcp_code_search/database.rb

@@ -0,0 +1,96 @@
+require "active_record"
+require "neighbor"
+require "digest"
+require "fileutils"
+
+module RailsMcpCodeSearch
+  module Database
+    DATA_DIR = File.join(ENV.fetch("XDG_DATA_HOME", File.join(Dir.home, ".local", "share")), "rails-mcp-code-search")
+
+    class << self
+      def setup(project_path: Dir.pwd, db_path: nil)
+        path = db_path || db_path_for(project_path)
+        ensure_data_directory(File.dirname(path))
+
+        Neighbor::SQLite.initialize!
+
+        ActiveRecord::Base.establish_connection(
+          adapter: "sqlite3",
+          database: path,
+          pool: 5,
+          timeout: 5000
+        )
+
+        configure_pragmas
+        create_schema
+        Metadata.set "project_path", project_path
+
+        File.chmod 0o600, path if File.exist?(path)
+
+        path
+      end
+
+      def db_path_for(project_path)
+        digest = Digest::SHA256.hexdigest(project_path)
+        File.join(DATA_DIR, "#{digest}.db")
+      end
+
+      private
+
+      def ensure_data_directory(dir)
+        FileUtils.mkdir_p dir, mode: 0o700
+        noindex = File.join(dir, ".noindex")
+        FileUtils.touch noindex unless File.exist?(noindex)
+      end
+
+      def configure_pragmas
+        ActiveRecord::Base.connection.execute "PRAGMA journal_mode=WAL"
+        ActiveRecord::Base.connection.execute "PRAGMA synchronous=NORMAL"
+        ActiveRecord::Base.connection.execute "PRAGMA cache_size=-64000"
+      end
+
+      def create_schema
+        ActiveRecord::Schema.define do
+          unless table_exists?(:chunks)
+            create_table :chunks do |t|
+              t.text :file_path, null: false
+              t.integer :line_start, null: false
+              t.integer :line_end, null: false
+              t.text :chunk_type, null: false
+              t.text :qualified_name
+              t.text :content, null: false
+              t.text :checksum, null: false
+              t.binary :embedding
+              t.integer :hit_count, default: 0
+              t.timestamps
+            end
+
+            add_index :chunks, :file_path
+            add_index :chunks, :checksum
+            add_index :chunks, :hit_count, order: { hit_count: :desc }
+          end
+
+          unless table_exists?(:metadata)
+            create_table :metadata, id: false do |t|
+              t.text :key, null: false, primary_key: true
+              t.text :value, null: false
+            end
+          end
+        end
+      end
+    end
+
+    class Metadata < ActiveRecord::Base
+      self.table_name = "metadata"
+      self.primary_key = "key"
+
+      def self.get(key)
+        find_by(key:)&.value
+      end
+
+      def self.set(key, value)
+        upsert({ key:, value: value.to_s }, unique_by: :key)
+      end
+    end
+  end
+end

data/lib/rails_mcp_code_search/embeddings/adapter.rb

@@ -0,0 +1,13 @@
+module RailsMcpCodeSearch
+  module Embeddings
+    class Adapter
+      def embed(texts)
+        raise NotImplementedError
+      end
+
+      def dimensions
+        raise NotImplementedError
+      end
+    end
+  end
+end

data/lib/rails_mcp_code_search/embeddings/local_adapter.rb

@@ -0,0 +1,29 @@
+require "informers"
+
+module RailsMcpCodeSearch
+  module Embeddings
+    class LocalAdapter < Adapter
+      DIMENSIONS = 384
+      MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+
+      def initialize
+        @mutex = Mutex.new
+      end
+
+      def embed(texts)
+        texts = Array(texts)
+        pipeline.call(texts)
+      end
+
+      def dimensions = DIMENSIONS
+
+      private
+
+      def pipeline
+        @mutex.synchronize do
+          @_pipeline ||= Informers.pipeline("embedding", MODEL)
+        end
+      end
+    end
+  end
+end

data/lib/rails_mcp_code_search/embeddings/openai_adapter.rb

@@ -0,0 +1,59 @@
+require "openai"
+
+module RailsMcpCodeSearch
+  module Embeddings
+    class OpenaiAdapter < Adapter
+      DIMENSIONS = 1536
+      MODEL = "text-embedding-3-small"
+      MAX_RETRIES = 3
+      MAX_CHARS = 15_000
+
+      ApiKeyError = Class.new(StandardError)
+
+      def initialize
+        api_key = ENV["RAILS_MCP_CODE_SEARCH_OPENAI_API_KEY"]
+        raise ApiKeyError, "RAILS_MCP_CODE_SEARCH_OPENAI_API_KEY environment variable is required for OpenAI provider" unless api_key
+
+        @client = OpenAI::Client.new(access_token: api_key)
+        @warned = false
+      end
+
+      def embed(texts)
+        texts = Array(texts).map { truncate(_1) }
+        warn_once
+
+        retries = 0
+        begin
+          response = @client.embeddings(parameters: { model: MODEL, input: texts })
+          response.dig("data").sort_by { _1["index"] }.map { _1["embedding"] }
+        rescue Faraday::TooManyRequestsError
+          retries += 1
+          raise if retries > MAX_RETRIES
+          sleep(2**retries)
+          retry
+        rescue => e
+          raise sanitize_error(e)
+        end
+      end
+
+      def dimensions = DIMENSIONS
+
+      private
+
+      def warn_once
+        return if @warned
+        @warned = true
+        $stderr.puts "[rails-mcp-code-search] WARNING: Source code from this repository will be sent to OpenAI's embedding API."
+      end
+
+      def truncate(text)
+        text.size > MAX_CHARS ? text[0, MAX_CHARS] : text
+      end
+
+      def sanitize_error(error)
+        message = error.message.gsub(/sk-[A-Za-z0-9_-]+/, "[REDACTED]")
+        StandardError.new(message)
+      end
+    end
+  end
+end

data/lib/rails_mcp_code_search/erb_parser.rb

@@ -0,0 +1,89 @@
+require "herb"
+
+module RailsMcpCodeSearch
+  class ErbParser
+    Result = RubyParser::Result
+    MIN_LINES = 3
+    MAX_CHUNKS = 30
+
+    def self.parse(source, file_path: nil)
+      new.parse(source, file_path:)
+    end
+
+    def parse(source, file_path: nil)
+      result = Herb.parse(source)
+      return SlidingWindowParser.parse(source, file_path:) unless result.success?
+
+      visitor = Visitor.new(source)
+      visitor.visit(result.value)
+      chunks = deduplicate(visitor.chunks).first(MAX_CHUNKS)
+
+      chunks.empty? ? SlidingWindowParser.parse(source, file_path:) : chunks
+    rescue => _e
+      SlidingWindowParser.parse(source, file_path:)
+    end
+
+    private
+
+    def deduplicate(chunks)
+      chunks.reject do |chunk|
+        chunks.any? { _1 != chunk && _1.line_start <= chunk.line_start && _1.line_end >= chunk.line_end }
+      end
+    end
+
+    class Visitor < Herb::Visitor
+      attr_reader :chunks
+
+      def initialize(source)
+        super()
+        @source = source
+        @lines = source.lines
+        @chunks = []
+      end
+
+      def visit_erb_block_node(node)
+        add_chunk node, "erb_block"
+        super
+      end
+
+      def visit_erb_if_node(node)
+        add_chunk node, "erb_conditional"
+        super
+      end
+
+      def visit_erb_unless_node(node)
+        add_chunk node, "erb_conditional"
+        super
+      end
+
+      def visit_erb_case_node(node)
+        add_chunk node, "erb_conditional"
+        super
+      end
+
+      def visit_html_element_node(node)
+        add_chunk node, "html_element"
+        super
+      end
+
+      private
+
+      def add_chunk(node, type)
+        line_start = node.location.start.line
+        line_end = node.location.end.line
+        return if (line_end - line_start + 1) < MIN_LINES
+
+        content = @lines[(line_start - 1)..(line_end - 1)]&.join
+        return if content.nil? || content.strip.empty?
+
+        @chunks << Result.new(
+          content:,
+          line_start:,
+          line_end:,
+          chunk_type: type,
+          qualified_name: nil
+        )
+      end
+    end
+  end
+end