gemchat 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3d1cb20bde07c3cbb881a02a2fe6caace2e8a464a13e53b1471439b038bf35c8
+  data.tar.gz: 92dee0546046145083489dbf54e1690849f92cd756f909ed976f4afa395526e7
+SHA512:
+  metadata.gz: 59cb4d314bede1b9b453bc75e477dd2f1c2c4667638df955bef65ebec95301371c1e5adca5f0ab4324338937de6715fd6ffcb8550caabef8773f9a4a53fd60e6
+  data.tar.gz: d0eaa1e6ca369a7da8fc5154ca8287250b41bd39262db96cc0fef70f9c30c5ab114e4cbe47e3ef638f8c9f144f1a232acda462ca1ddb3ebb686c51ab34800b3d

data/AGENTS.md

@@ -0,0 +1,354 @@
+# AGENTS.md
+
+Guidance for coding agents working in this repository.
+
+## What this is
+
+`gemchat` is a standalone Ruby gem and CLI. It indexes the documentation for the
+gems in the current project's own `Gemfile.lock` and serves BM25 search over it,
+offline, with no account and no API key.
+
+It is a **local** tool. The hosted `gemchat.org` service is a separate,
+explicitly opt-in tier — never a silent fallback. See
+`docs/plans/gemchat-local-bundle-index.md` for the full design; that plan is the
+source of truth for anything not settled here.
+
+This repository is **not** the Rails app. The app lives separately, and changes
+there should not land here.
+
+## Commands
+
+```sh
+bin/setup                      # or: bundle install
+bundle exec rake test          # Minitest
+bundle exec rake               # test + standard, the default task
+bundle exec standardrb lib exe test
+```
+
+Smoke test against a throwaway lockfile:
+
+```sh
+D=$(mktemp -d); export GEMCHAT_HOME="$D"
+printf 'GEM\n  remote: https://rubygems.org/\n  specs:\n    rake (13.4.2)\n\nPLATFORMS\n  ruby\n\nDEPENDENCIES\n  rake\n\nBUNDLED WITH\n   4.0.21\n' > "$D/Gemfile.lock"
+(cd "$D" && bundle exec --gemfile=$OLDPWD/Gemfile $OLDPWD/exe/gemchat index)
+(cd "$D" && bundle exec --gemfile=$OLDPWD/Gemfile $OLDPWD/exe/gemchat search "Rake::Task#enhance")
+rm -rf "$D"
+```
+
+Always set `GEMCHAT_HOME` when testing. The default root is the developer's real
+`~/.gemchat`.
+
+## Trying the gem locally
+
+Auto-index is opt-in by running `gemchat init` once in the project: that adds
+both Gemfile lines, grants trust and writes `.gemchat.yml`. (`gemchat index`
+alone also grants trust, for people who prefer to wire it up by hand.) After that, any `bundle install`/`add`/`update`
+that moves the lockfile digest indexes automatically and says one line; anything
+else is completely silent.
+
+
+To exercise the packaged gem and the Bundler plugin against real installs —
+rather than the library in-process — use the sandbox:
+
+```sh
+bin/sandbox              # temp project, prints the commands to drive it
+bin/sandbox keep         # leave it at tmp/sandbox so you can poke around
+bin/sandbox keep --reset # wipe and recreate
+```
+
+It writes a Gemfile that declares gemchat twice, as a dependency and as a
+plugin, both `path:`-sourced at this checkout, and runs `bundle install`. Then
+`gemchat index`, `gemchat search`, `bundle add` and `bundle update` all work in
+there, and the plugin hook fires on the last two.
+
+Two details that matter, and cost real time to find:
+
+- **The Gemfile has no `source` line.** With one, Bundler fetches remote
+  metadata for gems it does not need, and the sandbox needs the network.
+  Without one, a full install is well under a second and offline.
+- **`bin/sandbox` must be run directly, not under `bundle exec`.** Bundler
+  injects `BUNDLE_GEMFILE`, `BUNDLE_LOCKFILE`, `RUBYOPT` and `RUBYLIB` into its
+  children, and a subprocess inherits them — so `bundle install` then installs
+  the *caller's* bundle into the sandbox instead of the sandbox's own. It exits
+  0, writes no lockfile, and installs no plugin, so it looks like it worked.
+
+`test/support/sandbox.rb` is the programmatic form of this, used by the
+lifecycle tests. Getting the child environment right needs **all three** of:
+
+```ruby
+clean = ENV.to_h.reject { |k, _| k.match?(/\ABUNDLE|RUBYOPT|RUBYLIB|RUBYGEMS_/) }
+Open3.capture2e(clean, *argv, chdir: dir, unsetenv_others: true)
+```
+
+Two of these are easy to miss and each one fails silently:
+
+- **Stripping only `BUNDLE*` is not enough.** `RUBYOPT=-rbundler/setup` then
+  survives, `bundler/setup` runs *inside the child*, and it re-derives
+  `BUNDLE_GEMFILE` from the caller's directory. The child looks clean in the env
+  hash and is dirty in practice. `unsetenv_others: true` alone does not help,
+  because the variable is genuinely present to be inherited.
+- **`Open3` merges, it does not replace.** Omitting a variable does not unset it;
+  only `unsetenv_others: true` does. Re-merge the real `ENV` afterwards so
+  `PATH` survives.
+
+## Architecture
+
+| Path | Role |
+|---|---|
+| `lib/gemchat.rb` | requires, `Gemchat.root`, `Gemchat.config_home` |
+| `lib/gemchat/ri.rb` | ri store lookup, generation via RDoc, reading back to plain Ruby |
+| `lib/gemchat/chunker.rb` | ri structs → indexable chunks (one per method, one per class) |
+| `lib/gemchat/markdown.rb` | Markdown/RDoc → heading-scoped sections |
+| `lib/gemchat/prose.rb` | README, `guides/` and `doc/` discovery, one `source_type` each |
+| `lib/gemchat/store.rb` | SQLite schema, FTS5, BM25 search, vectors, hybrid fusion |
+| `lib/gemchat/symbols.rb` | the Prism pass: file:line and signature per method |
+| `skills/gemchat/SKILL.md` | the agent-facing skill; ships in the gem, guarded by `test/skill_test.rb` |
+| `lib/gemchat/stopwords.rb` | the 127-word list the `:any` lexical mode filters against |
+| `lib/gemchat/indexer.rb` | lockfile parsing and per-gem orchestration |
+| `lib/gemchat/lockfile.rb` | the sha256 that is the whole staleness check |
+| `lib/gemchat/manifest.rb` | `.gemchat.yml`: what the index was built from |
+| `lib/gemchat/trust.rb` | `trusted.json`: which projects may auto-index |
+| `lib/gemchat/hook.rb` | the plugin's logic, reached through the bundled copy |
+| `lib/gemchat/init.rb` | `gemchat init`: the only command that may touch a Gemfile |
+| `lib/gemchat/plugin_index.rb` | reads Bundler's record of which hooks are armed |
+| `lib/gemchat/errors.rb` | `Gemchat::Error`, in its own file so subclasses can be required first |
+| `lib/gemchat/cli.rb` | `init`, `index`, `reindex`, `search`, `vsearch`, `query`, `embed`, `status`, `version` |
+| `lib/gemchat/models.rb` | model registry, pinned digest, streamed download with resume |
+| `lib/gemchat/embedder.rb` | Null/Gguf/Supervised embedders, batching, backoff, the child |
+| `lib/gemchat/env.rb` | boolean env parsing |
+| `plugins.rb` | the Bundler entry point — `add_hook` and nothing else |
+
+Data flow: `Gemfile.lock` → `Indexer#lockfile_specs` → `Ri` (generate or reuse)
+→ `Chunker.from_ri` → `Symbols` stamps each chunk with its defining file and
+line → `Store#replace_gem` → FTS5 → `Store#search`.
+
+ri says a method exists; only the Prism pass says where. That is the one thing a
+search result cannot be used without and cannot be inferred from, and it is
+printed after every ri hit. Matched on (class, method, method_type) because that
+is all either side knows; an unmatched chunk keeps a nil location rather than
+borrowing someone else's path.
+
+Vectors are a parallel, entirely optional path. `gemchat embed` →
+`Store#unembedded_chunks` → `Embedder::Supervised` (child process) →
+`Store#store_embeddings` → float32 BLOBs; `gemchat vsearch` → one embedding →
+`Store#vector_search` → `dot()`. `gemchat query` fuses both by reciprocal rank via
+`Store#hybrid_search`. Nothing in the `search` path touches either, and
+`gemchat search` must keep working with no model file on disk at all.
+
+`GEMCHAT_HOME` (data) and `GEMCHAT_CONFIG_HOME` (settings, trust) are separate on
+purpose: wiping the cache must not discard trust decisions. Both live in
+`Gemchat::Paths`, its own file so the plugin shim can resolve one without
+requiring the whole gem.
+
+## Hard-won constraints
+
+These were each measured or discovered the hard way. Re-deriving them wastes a
+cycle, and several look like reasonable ideas until they are not.
+
+- **Plain Ruby only. No ActiveSupport.** It would add 13 gems
+  (`concurrent-ruby`, `i18n`, `tzinfo`, `drb`, `connection_pool`, …) to a CLI that
+  needs none of them. No `presence`, `compact_blank`, `squish`, or
+  `String#first`.
+- **RDoc refuses to write into an existing output directory.** Create only the
+  parent and leave the target alone.
+- **`RDoc::RI::Driver` takes `--doc-dir`**, not `--ri`/`--op`.
+- **Generate with `Dir.chdir(gem_dir)` and relative source dirs** (`lib`, `ext`).
+  Absolute paths make RDoc derive wrong class names.
+- **`SystemStackError` descends from `Exception`, not `StandardError`.** RDoc
+  overflows the stack on symlink cycles; a bare `rescue` misses it.
+- **FTS5 parses `:` as a column filter.** An unquoted `Puma::Server` fails with
+  "no such column". Every term must be quoted — see `Store.fts_query`. This is a
+  systematic collision with Ruby naming, not a one-off.
+- **`search` is an exact-phrase matcher, not a keyword search.** Quoting each
+  term escapes `:` but joining the quoted terms with a space is FTS5's
+  AND-of-a-phrase. Measured on a ~22k-chunk corpus: **0 of 5** known-answer
+  questions located, one exact hit for an identifier. So it is precise and has
+  no recall — keep that in mind before "fixing" it.
+- **OR-ing the terms is not the fix, and neither is dropping stopwords.** Four
+  combinations of FTS5's boolean semantics were measured on known-answer
+  queries: phrase 0/5, OR 1/5, OR-with-stopwords 1/5, AND 0/5. The reason is not
+  common-word noise — it is that a rare term in a short ri signature outranks
+  the prose that answers the question, because BM25 length normalisation rewards
+  exactly that shape. A paraphrase question and its answer share no vocabulary,
+  so no amount of term selection fixes it. See §11.3 of the plan.
+- **Source priors measured worse than none, so there are none.** Reusing the
+  app's 1.15 prose prior, 1.30, and a 1.15 ri prior against an evenly-split
+  ri/prose eval: hit@1 6/10 with no prior, 6/10 at 1.15 prose, 5/10 at 1.30, 4/10
+  at ri 1.15 — and MRR fell in every case (0.744 → 0.705 → 0.655 → 0.547) while
+  recall@10 stayed 10/10. A prior is blunt: it shifts a whole class, and cannot
+  tell a good prose chunk from a bad one. Do not add one without re-measuring.
+- **The hosted tier is a row API, not MCP.** The MCP endpoint is a plain JSON-RPC
+  POST — the transport was never the problem — but `query_docs` returns *Markdown*,
+  which a CLI cannot parse without breaking silently. So the client speaks
+  `POST /api/v1/query`; MCP stays the agent surface, and both call one
+  `HostedQuery` in `gemchat_app` so the ranking exists once.
+- **The hosted tier never falls back.** No route leads from a failed, unconfigured
+  or `lex:` hosted query to a local answer. A user who asked for hosted results and
+  got local ones cannot tell the difference, so that is a privacy bug.
+- **This bites fusion harder than search.** RRF trusts rank, so a bad arm gets
+  promoted rather than ignored. The lexical arm is for identifiers, where query
+  terms appear literally; the vector arm is for questions. Do not wire `:any` in
+  as the default without re-running the eval in §11.4, and do not try to fix the
+  lexical arm with term selection — that has been tried and falsified.
+- **`results_as_hash` gives every row integer keys too.** `{**row}` leaks `0, 1,
+  2…` into results and mixes symbol and string key styles. Select an explicit
+  column list instead.
+- **sqlite3 2.x discards a UDF's return value.** It reads `FunctionProxy#result`;
+  the block's own return value is thrown away. Returning the float from a
+  `create_function` block makes every rank `NULL`, which sorts as `0.0` — so
+  `vsearch` returns an arbitrary slice of the corpus in a plausible-looking
+  order. Assign `fp.result =` instead. This is a silent failure: no error, and
+  results that look like output.
+- **A vector query costs ~915ms at 22k chunks, not milliseconds.** Measured on a
+  real 22,141-chunk index: 20s to index, 494s to embed (99.97% success), 915ms per
+  vector search, 13.5MB store. The bottleneck is the UDF, which unpacks two
+  3072-byte blobs per row -- ~34M array elements a query. The 236-chunk index
+  reports ~5ms, which is how the wrong number survived in the plan.
+- **Narrowing candidates in SQL IS expressible** -- a candidate CTE joined to
+  `chunks` limits which rows the UDF runs on, so two-stage retrieval is one
+  statement, not something that has to be materialised in Ruby first. Measured:
+  915ms -> 22ms at a pool of 500, with the top-1 result identical on 8 of 8
+  questions and recall@200 of the full-scan top-200 at 48.8%. Deliberately not
+  enabled: the candidate generator is lexical, so a paraphrase sharing no
+  vocabulary returns nothing where a full scan would have found it. See 11.1.
+- **rllama aborts the process instead of raising.** SIGILL descends from
+  nothing `StandardError` catches, so in-process there is no way to detect,
+  retry, or degrade. Measured intermittently: the same command failed four times
+  running and later passed 20 of 20 unchanged, and `GGML_METAL_DISABLE=1` did
+  not help. Embedding runs in a supervised child, retried once with Metal off.
+- **The native embedder has a size cliff whose position cannot be pinned.** Real
+  prose fails at 1665 bytes and passes at 1634, while 2500 bytes of `"word "`
+  is fine and stripping to pure ASCII does not help. Do not tune to a measured
+  number: keep the 1200-byte per-item ceiling, the 16KB per-batch budget, and
+  the halve-and-retry backoff, and treat all three as defence in depth.
+- **The download 302s to a CDN, and each hop needs its own connection.**
+  Replaying the redirect target down the origin's open TLS session produces a 403
+  on a perfectly good signed URL. `use_ssl` also has to follow the scheme, or a
+  plain-HTTP mirror is unreachable. A 200 answer to a `Range` request means the
+  whole file is coming — rewind rather than append, or you get the right length
+  and the wrong bytes.
+- **`Models.present` is advisory.** It explains a missing model and returns
+  false. Treating that false as "cannot continue" makes `gemchat embed` exit 1 on
+  a clean machine and never fetch anything. Nothing stands between a missing
+  model and a downloaded one except that call.
+- **`unembedded_chunks` only returns rows where the vector `IS NULL`.** So a
+  model change cannot be reconciled by re-running `embed`; `embed --force` has
+  to clear the vectors, and the mismatch error has to name a command that does.
+- **Nomic needs task prefixes**, and omitting them is a quality loss rather than
+  a crash: the correct hit's similarity measured 0.515 unprefixed and 0.580
+  prefixed. They are part of the stored identity string for that reason.
+- **`define_singleton_method` rebinds `self` to the receiver.** A lambda installed
+  as a `with_stub` cannot call helper methods on the test instance. Bind what it
+  needs up front: `basis = method(:unit)`.
+- **`benchmark` is not a default gem in Ruby 4.0.** Use
+  `Process.clock_gettime(Process::CLOCK_MONOTONIC)`.
+- **Minitest 6 removed `Object#stub` and ships no `minitest/mock`.** Use the
+  `with_stub` helper in `test/test_helper.rb`.
+- **`Gemchat.root` must not be memoised.** A value derived from the environment
+  that is cached means the first reader wins for the life of the process; setting
+  `GEMCHAT_HOME` later silently does nothing.
+- **Only `lib/` and `ext/` are documented, and `--all` is deliberately omitted.**
+  `--all` adds 3–46% more chunks of private and internal machinery.
+- **Bundler does not generate ri on install; `gem install` does.** So ri
+  generation is the main path, not a fallback.
+- **A malformed `plugins.rb` fails the install**: exit 29, no `Gemfile.lock`
+  written, even though the gems themselves are already on disk. Caught by
+  raising, by a bad event name, and by the file being absent.
+- **`plugins.rb` must live in the gem, never the project root.** A project-root
+  `plugins.rb` is silently never executed — Bundler only ever loads
+  `<installed gem>/plugins.rb`. Same filename, opposite outcome, and it looks
+  installed while doing nothing.
+- **A `plugins.rb` that only runs, without `add_hook`, fires once and never
+  again.** Bundler records *registered hooks* in `.bundle/plugin/index` and
+  gates later calls on that record, so `bundle add`/`update` go silent.
+- **Root-scope every Bundler reference in `plugins.rb` as `::Bundler`.** Bundler
+  loads it with `load(path, true)` (anonymous wrapper) mid-flight inside
+  `module Bundler::Plugin`; a bare constant can raise `NameError` and kill the
+  install.
+- **Never gate the hook on `RAILS_ENV`.** Copied from Hyperdrive, whose host is a
+  Rails engine. gemchat is not a Rails tool, so in any plain Ruby project
+  `RAILS_ENV` is unset and the hook would be permanently silent.
+- **The shim's logic lives in `lib/`, never in `plugins.rb`.** Dual declaration
+  installs two copies and nothing pins them, so anything implemented in the shim
+  exists twice and can disagree with the version the project depends on.
+- **Use `require_relative` for sibling files.** Under a `path:`-sourced checkout
+  `lib/` is *not* on `$LOAD_PATH`, so `require "gemchat/paths"` fails while
+  `require_relative "paths"` works. Mixing the two broke the plugin end to end.
+- **The hook must never propagate an exception**, including from parsing the
+  subprocess output. It also has to be robust to `gemchat` not being on `PATH`
+  under `bundle install` — it runs the *bundled* `exe/gemchat` directly.
+- **`doc/` in an installed gem is hand-written prose, not generated ri.** An
+  earlier version of this file claimed the opposite and it cost real content:
+  measured across every installed gem with a `doc/`, 12 gems, 96 files, **83
+  prose, 0 generated ri**. `irb/Index.md` (23KB), `irb/EXTEND_IRB.md`,
+  `rake/glossary.rdoc` and `test/how-to.md` are all there and none are
+  derivable from method comments. Do not exclude it. What *is* excluded is the
+  generated shape specifically — `doc/**/ri/**` and rdoc's HTML/JS/CSS tree.
+- **A `#` inside a fenced code block is not a heading.** Without tracking fence
+  state, a comment in a README example splits the section that introduces it.
+- **Long prose sections are split, not truncated.** `MAX_BODY_CHARS` is tuned for
+  ri method bodies; applied to prose it dropped 13% of `doc/` content, worst on
+  files like `tutorial.rdoc` (22%), cutting mid-sentence.
+- **`readme`, `guides` and `doc` are separate `source_type`s, grouped only at
+  query time.** That is `gemchat_app`'s vocabulary: it gives `guides` a 1.15
+  ranking prior and derives a `readme_guides` group via `source_group` purely for
+  prose diversity. Folding `guides/` into `readme` would make local `guides` mean
+  something different from hosted `guides`.
+- **Bundler 4 writes no `PLUGINS` lockfile section**, so nothing pins the plugin
+  copy. Dual `gem` + `plugin` declaration installs two physical copies — but only
+  for a *published* gem; a `path:` source is referenced in place, so a local
+  harness cannot reproduce it.
+- **`after-install-all` takes a string**, and it does fire on `bundle install`,
+  `bundle add`, and `bundle update` — all three measured.
+- **The plugin failure mode is silence, not an error.** `test/plugin_shim_test.rb`
+  asserts each of these mechanically, because all of them are invisible in review.
+- **To test the plugin locally**, use a path source —
+  `gem "gemchat", path: …` / `plugin "gemchat", path: …`.
+  `bundle config set --local local.gemchat …` does not work for an unpublished
+  gem, and `bundle config set --local path …` sets the install location, which
+  breaks unrelated gems. Full details in §15.6a.
+
+## Conventions
+
+- Standard Ruby (standardrb), not rubocop. `ruby_version: 3.2` in
+  `.standard.yml` — write for 3.2, not for the local interpreter.
+- Minitest, one file per component, `test/test_helper.rb` holds shared fixtures.
+- Tests must not depend on a specific gem being installed. `IndexerTest::REAL_GEM`
+  discovers one from the running bundle; prefer that pattern over hardcoding.
+- Every test that touches the filesystem must set `GEMCHAT_HOME` to a tmpdir.
+- Do not commit `Gemfile.lock` changes you did not intend.
+- Comments explain *why*, especially where a workaround looks wrong. Do not
+  restate what the code does.
+
+## Commit style
+
+Subject line in the imperative, prefixed with the phase or theme, then a body
+that explains what was measured or discovered and why — not a list of changed
+files. Recent examples:
+
+- `Phase 1 spike: measure ri cost, settle plugin mechanics, working index+search`
+- `Correct the rubydex rationale: the cost is musl-specific, not universal`
+- `gemchat embed and vsearch: vectors in a supervised child, or not at all`
+
+## Status
+
+Working today: `init`, `index`, `reindex`, `search`, `vsearch`, `query`,
+`embed`, `status`, `version` (with `--hosted`) over ri plus README/`guides/`/`doc/` prose, a
+trust-gated, sha256-gated auto-index plugin, local vectors from an
+auto-downloaded `nomic-embed-text-v1.5`, RRF fusion in `query`, Prism source
+locations on every ri hit, and `skills/gemchat/SKILL.md`.
+
+Nothing in §20 is outstanding. The hosted tier (§12) shipped: `query --hosted`
+speaks `POST /api/v1/query` for rows, not MCP for prose, and never falls back to
+the local index.
+
+Two things were investigated and deliberately not built, so they do not get
+re-attempted by someone assuming they were simply forgotten:
+
+- **A keyword arm for the fusion's lexical side.** The current one is an
+  exact-phrase matcher that scores **0 of 5** on known-answer questions;
+  OR-ing the terms scores **1 of 5** and dropping stopwords does not move it,
+  because the failure is a vocabulary mismatch between a paraphrase and its
+  answer rather than term selection. Term selection was tried and falsified —
+  see the two FTS5 bullets above and §11.4.
+- **Source priors.** Measured worse than none; see the bullet above.

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## [Unreleased]
+
+## [0.1.0] - 2026-09-26
+
+- Initial release

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,10 @@
+# Code of Conduct
+
+"gemchat" follows [The Ruby Community Conduct Guideline](https://www.ruby-lang.org/en/conduct) in all "collaborative space", which is defined as community communications channels (such as mailing lists, submitted patches, commit comments, etc.):
+
+* Participants will be tolerant of opposing views.
+* Participants must ensure that their language and actions are free of personal attacks and disparaging personal remarks.
+* When interpreting the words and actions of others, participants should always assume good intentions.
+* Behaviour which can be reasonably considered harassment will not be tolerated.
+
+If you have any concerns about behaviour within this project, please contact us at ["zoras@users.noreply.github.com"](mailto:"zoras@users.noreply.github.com").

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Saroj Maharjan (zoras)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,144 @@
+# Gemchat
+
+Searches the documentation of the gems in **your own bundle**, offline, from a
+local index. No account, no API key, nothing leaves the machine.
+
+```console
+$ gemchat search "Rake::Task#enhance"
+ 1. [rake           13.4.2    ri     ] Rake::Task#enhance
+    enhance(deps=nil, &block)  (lib/rake/task.rb:115)
+
+$ gemchat query "how do I list all the available tasks"
+ 1. [rake           13.4.2    doc    ] rake › doc › Rake Command Line Usage · 1/3
+    The -T option displays the tasks...
+```
+
+## Why
+
+You are reading `sidekiq`'s source to answer a question, and the answer is in a
+README you do not have open, or in a method whose only documentation is a
+sentence in an `.rdb` file nobody has generated. gemchat indexes the
+documentation of every gem in your `Gemfile.lock` — generated ri, plus each gem's
+README, `guides/` and `doc/` — and searches it in milliseconds.
+
+It is a local tool. There is a hosted tier, and it is opt-in per invocation, but
+by default your queries never leave the machine.
+
+## Install
+
+```console
+$ gemchat init     # the only command that edits your Gemfile
+$ bundle install   # Bundler activates a newly declared plugin on the next install
+```
+
+`init` adds gemchat as a development dependency and as a Bundler plugin, and
+trusts the project. After that, any `bundle install`, `bundle add` or
+`bundle update` that moves your lockfile re-indexes automatically and prints one
+line. Anything else is silent — including on CI, and including in a repository
+you cloned, because you did not ask.
+
+## The three search commands
+
+They fail in **opposite** directions, so picking the wrong one returns nothing at
+all:
+
+| Question | Command |
+|---|---|
+| An exact identifier, an error string, a flag | `gemchat search` |
+| A natural-language question | `gemchat query` |
+| A paraphrase, vectors only | `gemchat vsearch` |
+
+`gemchat search` is BM25 but matches **exact phrases**. A question like *"how do I
+list all the available tasks"* returns no results, because those words do not
+appear together in any chunk. That is the design, not a broken index — reach for
+`query`.
+
+`gemchat query` fuses both engines by reciprocal rank, and takes a leading type:
+
+```console
+gemchat query "how do I list every task"         # both, fused (default)
+gemchat query "lex: Rake::Task#enhance"          # BM25 only, never loads a model
+gemchat query "vec: stop a process on a signal"  # vectors only
+```
+
+Always check `gemchat status` first. It says what is indexed, whether the index
+matches the current lockfile, and whether vectors exist:
+
+```
+index:     5 gems · 1095 chunks
+manifest:  up to date
+vectors:   not downloaded (146MB) → gemchat embed
+```
+
+## Semantic search
+
+`gemchat query` and `gemchat vsearch` need vectors, and **only `gemchat embed`
+downloads them** — a one-time 146MB fetch of `nomic-embed-text-v1.5`. They never
+download on their own, and they never silently answer from the local index
+instead; they tell you and stop.
+
+```console
+$ gemchat embed
+$ gemchat query "why does rake say 'Don't know how to build task'"
+```
+
+Every ri result carries a `path:line` from a Prism pass over the gem's source, so
+you can open the actual definition rather than trusting a summary.
+
+## Hosted tier (opt-in)
+
+`gemchat query --hosted` asks [gemchat.org](https://gemchat.org) instead of your
+local index. It needs a key — `GEMCHAT_API_KEY`, or a `credentials` file under
+the gemchat config home — and `gemchat init --hosted` records it as a project's
+default.
+
+It sends your lockfile's exact gem versions, and reports which ones it could
+scope to, so version fidelity is disclosed rather than assumed.
+
+What it costs you, stated plainly:
+
+- **Version drift is reduced, not eliminated.** A gem with no per-version index on
+  the server is answered from its full docs, which may not be your pinned release.
+  The response says so.
+- **The dependency graph leaves the machine.** That is the whole point of opting
+  in.
+- **A key and a network are required.** It is not zero-config.
+- **Different ranking.** PostgreSQL hybrid against a larger, cross-project index
+  versus local RRF over your bundle. Results will not be identical, and that is
+  expected.
+- **`search` and `vsearch` are local-only.** So is `lex:`, which is BM25.
+
+Nothing falls back: there is no route from a failed, unconfigured or `lex:` hosted
+query to a local answer, because you could not tell the difference.
+
+## What gets indexed
+
+| Source | `source_type` |
+|---|---|
+| ri generated from the installed gem's source | `ri` |
+| `README*` | `readme` |
+| `guides/` | `guides` |
+| `doc/` | `doc` |
+
+`doc/` is hand-written prose, not generated ri, and it is included. Generated
+ri under `doc/` and rdoc's HTML tree are excluded. Long prose sections are split
+at paragraph boundaries rather than truncated.
+
+## Development
+
+```console
+bin/setup              # or: bundle install
+bundle exec rake       # test + standardrb
+bin/sandbox            # a throwaway project with gemchat installed from this checkout
+```
+
+`bin/sandbox` is the fastest way to try a change against a real `bundle
+install`. Run it **directly, not under `bundle exec`** — see the note in
+`AGENTS.md`.
+
+`docs/plans/gemchat-local-bundle-index.md` is the design document and the source
+of truth; `AGENTS.md` records the constraints that are easy to get wrong.
+
+## License
+
+The gem is available as open source under the terms of the MIT License.

data/Rakefile

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "minitest/test_task"
+
+Minitest::TestTask.create
+
+require "standard/rake"
+
+# `rake standard` shells out to standardrb with no file list, which defaults to
+# ./ and so already covers plugins.rb. Verified with `--list-target-files`:
+# the top-level plugins.rb is executable code that ships to other people's
+# machines, so it must not fall outside the default task.
+task default: %i[test standard]