discord_store 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: beaeca859e4b101de87ff0fd0c779311606c36bf800f2ed766066bbe868f4847
-  data.tar.gz: 3b2dc1bead35992ed63ba051a32e7867b19bb0c6e50e3c02dd3977ab12fe5902
+  metadata.gz: f10dcdd64b9bd9de7e97d25e7d8281cc44a0810cb7a1925a638a89bfbba20fd6
+  data.tar.gz: 68b9779ba8f103aea3d34393fe82f6dadb745c6bda5e93a47d9231be9d5d39b2
 SHA512:
-  metadata.gz: 14516705bd0e8d8af2272b6383de32c9faa3497b3f3b3f9dc1468732c5fa84f34f77da5b9d50d81259354e804b93f0aa6fff38eea6e738c5a972d26eadb3132b
-  data.tar.gz: fe1d4aee25808ee3c4be6372e3e953c557f806812055a79eae16d57e2dad5624ae5637a80f0407479abce31e61c967061332fadb4dd0a2ff91e154a10b28c4c8
+  metadata.gz: 2e06b2344608460af2a506a210aa97f0d2ce28c8d80d911296ffcc4719d5d469389d1cbef201fcd6d3e7de6575bb06eb81263176a48934647deca3d1efe0b815
+  data.tar.gz: ebe9a1c48408f3a2a8a055ebd8bca5ab653ee0fa6a873a091198229b6efa277dbcca8c605d7d522ba82cdbe2863345bdc394552e27e40460c69476c50515a686

data/CHANGELOG.md

@@ -3,6 +3,40 @@
 All notable changes to this project are documented here. This project follows
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.2.0] - 2026-09-09
+
+### Added
+
+- **`DiscordStore::Search`** — `client.search`, a wrapper over
+  `GET /guilds/{id}/messages/search`. Discord does have a query API; this
+  exposes it deliberately and uses it for almost nothing, because it is
+  documented to under-return, is eventually consistent, caps a query at ten
+  thousand rows, and cannot see through AES-256-GCM anyway. SQLite remains the
+  query layer. Results come back as a `Result` rather than an Array, because an
+  Array invites the `.size` that Discord says not to trust.
+- **`BlobStore#orphans`** — chunks that no live manifest points at, the residue
+  of an interrupted write or a half-finished delete. Invisible to every other
+  method, because every other method starts from a manifest. This is the query
+  that justified the interface.
+- **`rake discord:orphans`**, and a search line in `rake discord:doctor` that
+  distinguishes "no intent" from "broken".
+- **`search_retry_floor`** configuration, so an unindexed guild answering
+  `retry_after: 0` does not become a hot loop.
+
+### Fixed
+
+- A comment in `blob_store.rb` claimed "Discord gives bots no search". It does;
+  the endpoint is documented and requires the `MESSAGE_CONTENT` privileged
+  intent. The README carried a softer version of the same error.
+
+### Security
+
+- Every search pins `author_id` to this application before the request leaves,
+  and filters the response again on the way in. Searching for another author
+  raises `SearchScopeError` while `own_messages_only` is on. Search reads across
+  a guild rather than a channel it was handed, which makes it the one endpoint
+  here that could turn a storage backend into a scraper.
+
 ## [0.1.0] - 2026-09-09
 
 First release.
@@ -57,4 +91,5 @@ Not verified against live Discord. The attachment ceiling, CDN `Range` support
 and real throughput are discovered at runtime rather than hardcoded, and none of
 them have been measured against a real guild.
 
+[0.2.0]: https://github.com/chayuto/discord_store/releases/tag/v0.2.0
 [0.1.0]: https://github.com/chayuto/discord_store/releases/tag/v0.1.0

data/README.md

@@ -51,9 +51,16 @@ def own_message?(message)
 end
 ```
 
-There is no message search, no member enumeration, no history export, and there
-will not be. The distance between a storage backend and a scraper is exactly
-whether it can read other people's messages, and this one cannot.
+There is no member enumeration and no history export, and there will not be.
+
+Discord *does* offer message search, and this library uses it in exactly one
+place — finding its own orphaned attachments. Every search pins `author_id` to
+this application, so the request cannot come back with somebody else's messages
+in the first place, and the results are filtered again on the way in. Asking for
+another author raises rather than being honoured.
+
+The distance between a storage backend and a scraper is exactly whether it can
+read other people's messages, and this one cannot.
 
 ---
 
@@ -168,6 +175,11 @@ CI runs every supported combination — ten of them — on every push.
 gem "discord_store"
 ```
 
+```sh
+gem install discord_store   # no Bundler, no Rails: the log and blob store
+                            # work on their own
+```
+
 The version number is an honest 0. Three values in here are discovered at
 runtime rather than hardcoded — the real attachment ceiling, whether the CDN
 honours HTTP `Range`, and sharded throughput — and none of them has been
@@ -352,6 +364,67 @@ a.value      # => 2
 
 ---
 
+## Discord's search API, and why reads still do not use it
+
+Discord has a query API. `GET /guilds/{id}/messages/search` takes a full-text
+`content` filter, channel and author filters, snowflake bounds, attachment
+filename and extension filters, and sorts by relevance or time. That is more of
+a query interface than a chat platform owes anybody, and it is worth being clear
+that this library's architecture is not a workaround for its absence.
+
+It is still not the read path, for four reasons that come from Discord's own
+documentation rather than from taste:
+
+1. **It is allowed to under-return.** *"Search may return slightly fewer results
+   than the limit specified"*, and clients *"should not rely on the length of
+   the messages array to paginate results"*. `total_results` may also be wrong
+   while messages are being written. An index that silently drops rows gives you
+   wrong answers, not slow ones — and a query layer that is occasionally wrong
+   is worse than no query layer, because you cannot tell which answers to doubt.
+2. **It is eventually consistent.** Fresh messages return `202` with
+   `{"code": 110000, "retry_after": n}` until indexed. There is no
+   read-your-writes.
+3. **It pages at 25 and cannot offset past 9975** — ten thousand rows per query,
+   reachable only in four hundred round trips.
+4. **Everything stored here is ciphertext.** Discord's index tokenises words;
+   AES-256-GCM output has none. Making payloads searchable would mean storing
+   them in the clear on somebody else's servers, which is a far larger
+   concession than the one on the front of this README.
+
+So it is used for exactly one thing, which a channel scan answers badly:
+
+```ruby
+client.blobs.orphans
+# => [{ message_id: "...", channel_id: "...", filename: "0.ds1", size: 8388608 }]
+```
+
+A blob is a manifest plus its chunks, written in that order. Interrupt the
+middle — a crash, a rate limit that outlived its retries — and the chunks are
+invisible to every other method here, because they all start from a manifest,
+while still counting against the guild forever. Search finds them by matching
+`.ds1`, an extension that was already public and says nothing about contents.
+
+```sh
+rake discord:orphans
+```
+
+Needs the `MESSAGE_CONTENT` privileged intent, which is off by default. Nothing
+else in the library needs it, and `rake discord:doctor` reports its absence as a
+missing capability rather than a failure.
+
+**The safety pin.** Search reads across a guild rather than a channel it was
+handed, which makes it the one endpoint here that could turn a storage backend
+into a scraper. Every search pins `author_id` to this application before the
+request leaves, so other people's messages cannot come back in the first place,
+and the response is filtered again on arrival. Asking for another author raises:
+
+```ruby
+client.search.messages(author_id: ["999..."])
+# => DiscordStore::SearchScopeError
+```
+
+---
+
 ## What does not work
 
 - **No server-side query.** Anything not in the local view means scanning a

data/lib/discord_store/blob_store.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require "digest"
+require "set"
 require "stringio"
 
 module DiscordStore
@@ -109,6 +110,47 @@ module DiscordStore
       manifest
     end
 
+    # Chunks that no live manifest points at.
+    #
+    # A blob is a manifest plus its chunks, written in that order and deleted in
+    # the other. Anything that interrupts the middle -- a crash, a rate limit
+    # that outlived its retries, a delete that got halfway -- leaves chunks with
+    # nothing referring to them. They are invisible to every other method here,
+    # because every other method starts from a manifest, and they count against
+    # the guild's storage forever.
+    #
+    # Finding them means asking a question no channel scan answers well: which
+    # attachments exist that nothing references. Discord's search can answer it
+    # directly, filtering on an extension that is already public and says
+    # nothing about the contents.
+    #
+    # Costs one search page per 25 chunks and needs the MESSAGE_CONTENT intent.
+    #
+    # @param search [Search, nil] defaults to one built from this config
+    # @return [Array<Hash>] each {message_id:, channel_id:, filename:, size:}
+    # @raise [MissingIntentError] if the privileged intent is not enabled
+    def orphans(search: nil)
+      finder = search || Search.new(rest: @rest, config: @config)
+      referenced = referenced_message_ids
+
+      found = []
+      finder.each(has: "file",
+                  attachment_extension: Search::EXTENSION,
+                  channel_ids: @config.blob_channel_ids) do |message|
+        next if referenced.include?(message["id"].to_s)
+
+        Array(message["attachments"]).each do |attachment|
+          # Spilled log records share the extension and are not blob chunks.
+          next if attachment["filename"].to_s == Codec::SPILL_FILENAME
+
+          found << { message_id: message["id"].to_s, channel_id: message["channel_id"].to_s,
+                     filename: attachment["filename"].to_s, size: attachment["size"].to_i }
+        end
+      end
+
+      found
+    end
+
     # @param key [String]
     # @return [String] the blob's bytes
     # @raise [NotFoundError]
@@ -367,6 +409,16 @@ module DiscordStore
       end
     end
 
+    def referenced_message_ids
+      @index.warm!
+      @index.keys.each_with_object(Set.new) do |key, ids|
+        manifest = @index.get(key)
+        next if manifest.nil?
+
+        manifest.chunks.each { |chunk| ids << chunk[:message_id].to_s }
+      end
+    end
+
     def safe_delete(channel_id, message_id)
       @rest.delete_message(channel_id, message_id)
     rescue APIError
@@ -375,10 +427,13 @@ module DiscordStore
 
     # Maps blob keys to manifests.
     #
-    # Discord gives bots no search, so a lookup by key would otherwise mean
-    # scanning a channel. This scans it exactly once, at first use, and keeps the
-    # result. Manifests are one small message per blob, so the scan is a hundred
-    # blobs per request.
+    # A lookup by key would otherwise mean scanning a channel. This scans it
+    # exactly once, at first use, and keeps the result. Manifests are one small
+    # message per blob, so the scan is a hundred blobs per request.
+    #
+    # Discord's message search could answer a key lookup directly, but only if
+    # the key were stored in plaintext, and a blob key names the thing it holds.
+    # Scanning once is cheaper than telling Discord what our files are called.
     #
     # Swap in your own if you would rather the index lived in Postgres: anything
     # answering get/put/delete/keys will do.

data/lib/discord_store/configuration.rb

@@ -23,6 +23,11 @@ module DiscordStore
       global_rate_limit: 45,
       quota_timeout: 15.0,
       max_retries: 5,
+      # Discord answers 202 with retry_after 0 when it has no better estimate,
+      # and says to "retry after a short delay". This is what short means.
+      # Tests drop it so the suite exercises the retry rather than sleeping
+      # through it, which is the same reason the fake's rate limit is settable.
+      search_retry_floor: 0.5,
       open_timeout: 5.0,
       read_timeout: 30.0,
       write_timeout: 30.0,
@@ -85,7 +90,7 @@ module DiscordStore
 
     attr_accessor :global_rate_limit, :quota_timeout, :max_retries,
                   :open_timeout, :read_timeout, :write_timeout,
-                  :user_agent, :api_base, :logger
+                  :user_agent, :api_base, :logger, :search_retry_floor
 
     # Refuse to read any message not authored by +application_id+. This is the
     # line between a storage backend and a scraper, and it is enforced in code

data/lib/discord_store/errors.rb

@@ -89,4 +89,47 @@ module DiscordStore
   # been written by somebody else. discord_store only ever reads its own bot's
   # messages; see DiscordStore::Transport::REST#assert_own_message!
   class ForeignMessageError < Error; end
+
+  # Discord's message index had not caught up with the messages being searched
+  # for, and did not catch up within the allowed number of retries.
+  #
+  # This is not a failure so much as a statement about what search is: an index
+  # maintained asynchronously beside the messages, not the messages themselves.
+  # A write is durable the moment create_message returns; it is *findable* some
+  # time after that, and Discord does not promise when.
+  class IndexNotReadyError < Error
+    attr_reader :documents_indexed
+
+    def initialize(message = nil, documents_indexed: nil)
+      @documents_indexed = documents_indexed
+      super(message || "Discord's search index is not ready for this guild yet")
+    end
+  end
+
+  # Searching requires the MESSAGE_CONTENT privileged intent, which Discord
+  # grants per application and reviews by hand once a bot is in 100 guilds.
+  class MissingIntentError < Error
+    def initialize(message = nil)
+      super(message || <<~MSG.strip)
+        Discord refused the search request. GET /guilds/{id}/messages/search is
+        gated on the MESSAGE_CONTENT privileged intent, which is off by default
+        and has to be enabled for the application in the Developer Portal, under
+        Bot -> Privileged Gateway Intents. Past 100 guilds Discord reviews the
+        request by hand, and "I am using it as a database" is not a use case it
+        approves.
+
+        Everything else in this library works without the intent. Search is the
+        only part that needs it.
+      MSG
+    end
+  end
+
+  # An attempt to search outside the messages this bot wrote, while the
+  # own_messages_only guard is on.
+  #
+  # Search is the one endpoint in Discord's API that could turn this library
+  # into a scraper: it reads across a whole guild rather than a channel this bot
+  # was pointed at. The guard is enforced by pinning author_id to our own
+  # application, and this is what you get for trying to unpin it.
+  class SearchScopeError < Error; end
 end

data/lib/discord_store/search.rb

@@ -0,0 +1,220 @@
+# frozen_string_literal: true
+
+module DiscordStore
+  # Discord's message index, exposed deliberately and used for almost nothing.
+  #
+  # Discord does have a query API. `GET /guilds/{id}/messages/search` takes a
+  # full-text `content` filter, channel and author filters, snowflake bounds,
+  # attachment filename and extension filters, and sorts by relevance or time.
+  # It is more of a query interface than a chat platform owes anybody.
+  #
+  # It is still not where this library reads from, for four reasons that are
+  # Discord's documentation rather than an opinion:
+  #
+  #   1. It is allowed to under-return. "Search may return slightly fewer
+  #      results than the limit specified", and clients "should not rely on the
+  #      length of the messages array to paginate". An index that silently drops
+  #      rows gives you wrong answers, not slow ones, and a query layer that is
+  #      sometimes wrong is worse than no query layer.
+  #   2. It is eventually consistent. Fresh messages answer 202 until indexed,
+  #      so there is no read-your-writes.
+  #   3. It pages at 25 and cannot offset past 9975, capping any single query
+  #      at ten thousand rows across four hundred round trips.
+  #   4. Everything this library stores is AES-256-GCM ciphertext. Discord's
+  #      index tokenises words; there are none. Making the payloads searchable
+  #      would mean storing them in the clear on somebody else's servers, which
+  #      is a much larger concession than the one on the front of the README.
+  #
+  # So SQLite remains the query layer, and this exists for the things search is
+  # genuinely better at than a channel scan: finding our own messages across a
+  # guild without knowing which channel they are in. The metadata it matches on
+  # -- the +DS1+ framing prefix and the +.ds1+ attachment extension -- is
+  # plaintext already, and deliberately says nothing about the contents.
+  #
+  # @see DiscordStore::BlobStore#orphans for the reason this was built
+  class Search
+    # The plaintext framing prefix every message this library writes begins
+    # with. Not a secret, and not informative: it identifies the format, not the
+    # contents.
+    MARKER = "DS1"
+
+    # Attachment extension used by both blob chunks and spilled log records.
+    EXTENSION = "ds1"
+
+    # One page. Discord's own maximum.
+    PAGE_SIZE = Transport::REST::SEARCH_PAGE_LIMIT
+
+    # A page of results.
+    #
+    # Deliberately not a bare Array: +total_results+ is documented as
+    # inaccurate while messages are being written, and the number of messages
+    # returned is documented as possibly fewer than asked for. Handing back an
+    # array would invite exactly the +.size+ that Discord says not to trust.
+    Result = Struct.new(:messages, :total_results, :indexing, keyword_init: true) do
+      include Enumerable
+
+      def each(&) = messages.each(&)
+
+      # @return [Integer] how many came back, which is not how many exist
+      def size = messages.size
+
+      def empty? = messages.empty?
+
+      # Always true. Kept as a method so calling code reads honestly.
+      #
+      # @return [Boolean]
+      def approximate? = true
+
+      # @return [Boolean] whether Discord is still backfilling this guild
+      def indexing? = !indexing.nil? && indexing
+    end
+
+    attr_reader :config
+
+    # @param rest [Transport::REST]
+    # @param config [Configuration]
+    # @param guild_id [String, nil] defaults to the configured guild
+    def initialize(rest:, config:, guild_id: nil)
+      @rest = rest
+      @config = config
+      @guild_id = (guild_id || config.guild_id)&.to_s
+
+      raise ConfigurationError, "guild_id is required to search" if @guild_id.nil? || @guild_id.empty?
+    end
+
+    # Searches for messages this bot wrote.
+    #
+    # @param content [String, nil] full-text filter; only ever matches the
+    #   plaintext framing, never a payload
+    # @param channel_ids [Array<String>, nil] restrict to these channels
+    # @param after [Time, String, Integer, nil] exclusive lower bound
+    # @param before [Time, String, Integer, nil] exclusive upper bound
+    # @param has [Array<String>, String, nil] e.g. "file"
+    # @param attachment_extension [String, nil]
+    # @param attachment_filename [String, nil]
+    # @param pinned [Boolean, nil]
+    # @param sort_by [String, nil] "timestamp" or "relevance"
+    # @param sort_order [String, nil] "asc" or "desc"
+    # @param limit [Integer]
+    # @param offset [Integer]
+    # @param author_id [Array<String>, nil] refused while own_messages_only is on
+    # @return [Result]
+    def messages(content: nil, channel_ids: nil, after: nil, before: nil, has: nil,
+                 attachment_extension: nil, attachment_filename: nil, pinned: nil,
+                 sort_by: nil, sort_order: nil, limit: PAGE_SIZE, offset: 0, author_id: nil)
+      params = {
+        content: content,
+        channel_id: Array(channel_ids).map(&:to_s),
+        min_id: bound(after),
+        max_id: bound(before),
+        has: Array(has),
+        attachment_extension: Array(attachment_extension),
+        attachment_filename: Array(attachment_filename),
+        pinned: pinned,
+        sort_by: sort_by,
+        sort_order: sort_order,
+        limit: limit.to_i.clamp(1, PAGE_SIZE),
+        offset: assert_offset!(offset),
+        author_id: scope_authors(author_id)
+      }
+
+      body = @rest.search_messages(@guild_id, **params)
+
+      Result.new(
+        messages: body["messages"],
+        total_results: body["total_results"],
+        indexing: body["indexing"]
+      )
+    end
+
+    # Every message this library wrote, anywhere in the guild.
+    #
+    # @return [Result]
+    def ours(**options) = messages(content: MARKER, **options)
+
+    # Every attachment this library uploaded: blob chunks and spilled records.
+    #
+    # @return [Result]
+    def attachments(**options)
+      messages(has: "file", attachment_extension: EXTENSION, **options)
+    end
+
+    # Pages through results.
+    #
+    # Pages by +offset+ rather than by the size of the last page, because
+    # Discord says not to paginate on that size. Stops at the offset ceiling
+    # with an explicit error rather than quietly returning a truncated set.
+    #
+    # @yieldparam message [Hash]
+    # @return [Enumerator, void]
+    def each(**options, &block)
+      return enum_for(:each, **options) unless block
+
+      offset = options.delete(:offset).to_i
+      seen = 0
+
+      loop do
+        page = messages(**options, offset: offset, limit: PAGE_SIZE)
+        page.each(&block)
+
+        seen += page.size
+        offset += PAGE_SIZE
+
+        break if page.empty? || seen >= page.total_results
+        break if exhausted?(offset, page)
+      end
+    end
+
+    private
+
+    def exhausted?(offset, page)
+      return false if offset <= Transport::REST::SEARCH_OFFSET_LIMIT
+
+      raise Error, <<~MSG.strip
+        Search cannot be offset past #{Transport::REST::SEARCH_OFFSET_LIMIT}, and this
+        query claims #{page.total_results} results. Narrow it with channel_ids or a
+        time range; there is no page #{offset}. This is a limit of Discord's index,
+        not of this library, and it is one of the reasons reads do not go through it.
+      MSG
+    end
+
+    # The safety pin.
+    #
+    # Search reads across a guild rather than a channel it was handed, so it is
+    # the one call in this library that could return somebody else's messages.
+    # Pinning author_id means the request cannot come back with them in the
+    # first place, rather than relying on filtering afterwards -- though
+    # {Transport::REST#normalise_search} filters afterwards as well.
+    def scope_authors(requested)
+      return Array(requested).map(&:to_s) unless config.own_messages_only
+
+      if requested && Array(requested).map(&:to_s) != [config.application_id.to_s]
+        raise SearchScopeError, <<~MSG.strip
+          Refusing to search for messages written by #{Array(requested).join(", ")}.
+          own_messages_only is on, which pins every search to this application's own
+          messages. Reading other people's messages is the line between a storage
+          backend and a scraper, and search is the endpoint that would cross it.
+          Set own_messages_only = false if you genuinely mean to, and know that
+          nothing else in this library will help you.
+        MSG
+      end
+
+      [config.application_id.to_s]
+    end
+
+    def assert_offset!(offset)
+      value = offset.to_i
+      return value if value <= Transport::REST::SEARCH_OFFSET_LIMIT
+
+      raise ArgumentError,
+            "offset #{value} exceeds Discord's maximum of #{Transport::REST::SEARCH_OFFSET_LIMIT}"
+    end
+
+    def bound(value)
+      return nil if value.nil?
+      return Snowflake.from_time(value).to_s if value.is_a?(Time)
+
+      value.to_s
+    end
+  end
+end

data/lib/discord_store/tasks.rake

@@ -1,5 +1,17 @@
 # frozen_string_literal: true
 
+# Reports whether search works, without making its absence look like a failure:
+# the MESSAGE_CONTENT intent is off by default and everything except orphan
+# detection works fine without it.
+def report_search(client)
+  result = client.search.ours(limit: 1)
+  puts "search:           available#{" (guild still indexing)" if result.indexing?}"
+rescue DiscordStore::MissingIntentError
+  puts "search:           unavailable (MESSAGE_CONTENT intent off; only rake discord:orphans needs it)"
+rescue DiscordStore::IndexNotReadyError
+  puts "search:           index not ready yet for this guild"
+end
+
 namespace :discord do
   desc "Rebuild the local database from the Discord log"
   task replay: :environment do
@@ -75,6 +87,29 @@ namespace :discord do
 
     puts "attachment limit: #{client.limits.attachment_limit} bytes"
     puts "chunk size:       #{client.limits.chunk_size} bytes"
+
+    report_search(client)
+  rescue DiscordStore::Error => e
+    abort "#{e.class}: #{e.message}"
+  end
+
+  desc "Find blob chunks that no manifest points at (needs MESSAGE_CONTENT)"
+  task orphans: :environment do
+    client = DiscordStore.client
+    found = client.blobs.orphans
+
+    if found.empty?
+      puts "no orphaned chunks"
+    else
+      bytes = found.sum { |o| o[:size] }
+      puts "#{found.size} orphaned chunk(s), #{bytes} bytes:"
+      found.each { |o| puts "  #{o[:channel_id]}/#{o[:message_id]} #{o[:filename]} (#{o[:size]}b)" }
+      puts
+      puts "These are chunks whose manifest never landed or was half-deleted."
+      puts "Nothing references them and nothing else will find them."
+    end
+  rescue DiscordStore::MissingIntentError => e
+    abort e.message
   rescue DiscordStore::Error => e
     abort "#{e.class}: #{e.message}"
   end

data/lib/discord_store/transport/fake.rb

@@ -36,6 +36,9 @@ module DiscordStore
       #   Permissive by default so that tests exercise logic rather than sleep;
       #   pass Discord's real per-channel window ({limit: 5, reset_after: 5.0})
       #   when the point of the test is the limiter itself.
+      SEARCH_PAGE_SIZE = 25
+      SEARCH_OFFSET_CEILING = 9975
+
       def initialize(application_id: "111111111111111111", clock: -> { Time.now },
                      rate_limit: { limit: 1000, reset_after: 0.05 })
         @application_id = application_id.to_s
@@ -47,6 +50,9 @@ module DiscordStore
         @requests = []
         @pending_rate_limits = 0
         @sequence = 0
+        @search_pending = 0
+        @search_under_returns = false
+        @search_denied = false
         @mutex = Mutex.new
       end
 
@@ -101,6 +107,32 @@ module DiscordStore
       # @return [Array<Hash>]
       def messages_in(channel_id) = @channels[channel_id.to_s]
 
+      # Makes the next +count+ searches answer 202 "index not yet available",
+      # which is what Discord does until it has indexed a guild.
+      #
+      # @return [void]
+      def delay_search_index(count: 1)
+        @search_pending = count
+      end
+
+      # Makes search return one fewer result per page than it should.
+      #
+      # Discord documents this: "search may return slightly fewer results than
+      # the limit specified". Code that paginates on the size of the returned
+      # array silently loses rows, so the fake does it too.
+      #
+      # @return [void]
+      def search_under_returns!
+        @search_under_returns = true
+      end
+
+      # Makes search answer 403, as it does without the MESSAGE_CONTENT intent.
+      #
+      # @return [void]
+      def deny_search!
+        @search_denied = true
+      end
+
       # @return [Integer] total API calls seen, for asserting on request counts
       def request_count = @requests.size
 
@@ -112,7 +144,10 @@ module DiscordStore
       def dispatch(request, uri)
         method = request.verb.to_s.upcase
         path = uri.path
-        query = URI.decode_www_form(uri.query.to_s).to_h
+        pairs = URI.decode_www_form(uri.query.to_s)
+        # .to_h keeps only the last value for a repeated key, which is exactly
+        # what array query params are, so search gets the pairs instead.
+        query = pairs.to_h
 
         case method
         when "POST"
@@ -134,14 +169,16 @@ module DiscordStore
             not_found
           end
         when "GET"
-          dispatch_get(path, query)
+          dispatch_get(path, query, pairs)
         else
           not_found
         end
       end
 
-      def dispatch_get(path, query)
+      def dispatch_get(path, query, pairs = [])
         case path
+        when %r{/guilds/(\d+)/messages/search\z}
+          search_messages(::Regexp.last_match(1), pairs)
         when %r{/channels/(\d+)/messages/(\d+)\z}
           get_message(::Regexp.last_match(1), ::Regexp.last_match(2))
         when %r{/channels/(\d+)/messages\z}
@@ -245,13 +282,141 @@ module DiscordStore
         Response.new(status: 204, headers: rate_limit_headers, body: nil)
       end
 
+      # Discord's message search, including the parts that make it unsuitable
+      # as a read path. A generous fake here would let code ship that breaks
+      # the first time the index lags or drops a row.
+      # +guild_id+ is unused: the fake holds one guild's worth of channels, and
+      # the parameter is kept so the signature matches the endpoint it models.
+      def search_messages(_guild_id, pairs)
+        return forbidden("Missing Access") if @search_denied
+
+        if @search_pending.positive?
+          @search_pending -= 1
+          return index_not_ready
+        end
+
+        params = group_params(pairs)
+        matches = search_candidates(params)
+        window = search_window(matches, params)
+
+        ok({
+             "total_results" => matches.size,
+             "doing_deep_historical_index" => false,
+             # Nested one deep: the shape Discord kept after it stopped
+             # returning the surrounding context of each hit.
+             "messages" => window.map { |message| [message] }
+           })
+      end
+
+      def search_candidates(params)
+        all = @channels.flat_map { |channel_id, messages| messages.map { |m| [channel_id, m] } }
+
+        all.select { |channel_id, message| search_match?(channel_id, message, params) }
+           .map { |_, message| message }
+           .sort_by { |message| -message["id"].to_i }
+      end
+
+      def search_match?(channel_id, message, params)
+        search_scope_match?(channel_id, message, params) &&
+          search_bounds_match?(message, params) &&
+          search_body_match?(message, params)
+      end
+
+      def search_scope_match?(channel_id, message, params)
+        authors = params["author_id"]
+        channels = params["channel_id"]
+
+        return false if authors.any? && !authors.include?(message.dig("author", "id"))
+        return false if channels.any? && !channels.include?(channel_id)
+
+        true
+      end
+
+      def search_bounds_match?(message, params)
+        id = message["id"].to_i
+        min = params["min_id"].first
+        max = params["max_id"].first
+
+        return false if min && id <= min.to_i
+        return false if max && id >= max.to_i
+
+        true
+      end
+
+      def search_body_match?(message, params)
+        content = params["content"]
+
+        return false if content.any? && !search_content_match?(message, content)
+        return false if params["has"].include?("file") && Array(message["attachments"]).empty?
+
+        search_attachment_match?(message, params)
+      end
+
+      def search_content_match?(message, needles)
+        haystack = message["content"].to_s.downcase
+        needles.all? { |needle| haystack.include?(needle.to_s.downcase) }
+      end
+
+      def search_attachment_match?(message, params)
+        extensions = params["attachment_extension"]
+        filenames = params["attachment_filename"]
+        return true if extensions.empty? && filenames.empty?
+
+        names = Array(message["attachments"]).map { |a| a["filename"].to_s }
+        return false if extensions.any? && names.none? { |n| extensions.include?(n.split(".").last) }
+        return false if filenames.any? && names.none? { |n| filenames.include?(n) }
+
+        true
+      end
+
+      def search_window(matches, params)
+        limit = (params["limit"].first || SEARCH_PAGE_SIZE).to_i.clamp(1, SEARCH_PAGE_SIZE)
+        offset = (params["offset"].first || 0).to_i
+        return [] if offset > SEARCH_OFFSET_CEILING
+
+        window = matches.slice(offset, limit) || []
+        # "Search may return slightly fewer results than the limit specified."
+        window = window[0...-1] if @search_under_returns && window.size > 1
+        window
+      end
+
+      def group_params(pairs)
+        grouped = Hash.new { |hash, key| hash[key] = [] }
+        pairs.each { |key, value| grouped[key] << value }
+        grouped
+      end
+
+      def index_not_ready
+        Response.new(
+          status: 202,
+          headers: rate_limit_headers.merge("content-type" => "application/json"),
+          body: JSON.generate({
+                                "message" => "Index not yet available. Try again later",
+                                "code" => 110_000,
+                                "documents_indexed" => 0,
+                                "retry_after" => 0
+                              })
+        )
+      end
+
+      def forbidden(message)
+        Response.new(
+          status: 403,
+          headers: rate_limit_headers.merge("content-type" => "application/json"),
+          body: JSON.generate({ "message" => message, "code" => 50_001 })
+        )
+      end
+
       def cdn_response(uri)
         segments = uri.path.split("/").reject(&:empty?)
         attachment_id = segments[2]
         record = @attachments[attachment_id]
         return not_found unless record
 
-        query = URI.decode_www_form(uri.query.to_s).to_h
+        pairs = URI.decode_www_form(uri.query.to_s)
+        # .to_h keeps only the last value for a repeated key, which is exactly
+        # what array query params are, so search gets the pairs instead.
+        query = pairs.to_h
         expires_at = query["ex"].to_s.to_i(16)
 
         # The whole point of the fake: an expired link 404s, exactly as Discord's

data/lib/discord_store/transport/rest.rb

@@ -9,15 +9,30 @@ module DiscordStore
     # The Discord REST surface this library uses, and nothing else.
     #
     # Deliberately small. Every method here exists to read or write data that
-    # this bot itself wrote. There is no message search, no member enumeration,
-    # no history export, and there will not be: the difference between a storage
-    # backend and a scraper is whether it can read other people's messages, and
-    # that difference is enforced in {#assert_own_message!} rather than in a
-    # paragraph of the README.
+    # this bot itself wrote. There is no member enumeration and no history
+    # export: the difference between a storage backend and a scraper is whether
+    # it can read other people's messages, and that difference is enforced in
+    # {#assert_own_message!} rather than in a paragraph of the README.
+    #
+    # {#search_messages} is the one endpoint here that reads across a guild
+    # rather than a channel it was handed, which makes it the obvious way to
+    # turn this library into the thing it is not. It is therefore pinned to our
+    # own author id before the request leaves, and filtered again on the way
+    # back. See DiscordStore::Search.
     class REST
       MESSAGE_PAGE_LIMIT = 100
       BULK_DELETE_LIMIT = 100
 
+      # Search pages at 25, not 100, and cannot be offset past 9975. Together
+      # those cap a single query at ten thousand results, reachable only 25 at a
+      # time -- four hundred round trips to exhaust one query.
+      SEARCH_PAGE_LIMIT = 25
+      SEARCH_OFFSET_LIMIT = 9975
+
+      # Discord answers 202 with this code when the guild is not indexed yet.
+      INDEX_NOT_READY_CODE = 110_000
+      SEARCH_INDEX_RETRIES = 5
+
       # Discord refuses to bulk-delete messages older than two weeks. Past that
       # the only route is one request per message, which is why the default
       # delete policy is to tombstone instead.
@@ -183,6 +198,53 @@ module DiscordStore
 
       # --- Guild and channel metadata ----------------------------------------
 
+      # --- Search ------------------------------------------------------------
+
+      # Searches a guild's messages.
+      #
+      # Three things about this endpoint are worth knowing before trusting it,
+      # and all three are Discord's own documentation rather than opinion:
+      #
+      #   * It is an index maintained beside the messages, not the messages. A
+      #     write is durable when create_message returns and findable some
+      #     unspecified time later; until then this answers 202.
+      #   * It is allowed to under-return. "Search may return slightly fewer
+      #     results than the limit specified", and total_results "may not be
+      #     accurate" while messages are being written. Results are a lower
+      #     bound, never a count.
+      #   * It needs the MESSAGE_CONTENT privileged intent.
+      #
+      # @param guild_id [String]
+      # @param params [Hash] query string params; array values are repeated
+      # @return [Hash] {"messages" => [Hash], "total_results" => Integer,
+      #   "indexing" => Boolean}
+      # @raise [IndexNotReadyError] the index never caught up
+      # @raise [MissingIntentError] the intent is not enabled
+      def search_messages(guild_id, **params)
+        route = "GET /guilds/#{guild_id}/messages/search"
+        attempt = 0
+
+        loop do
+          attempt += 1
+          body = search_request(guild_id, params, route)
+
+          return normalise_search(body) unless index_pending?(body)
+
+          if attempt > SEARCH_INDEX_RETRIES
+            raise IndexNotReadyError.new(
+              "#{route}: index still not ready after #{SEARCH_INDEX_RETRIES} retries",
+              documents_indexed: body["documents_indexed"]
+            )
+          end
+
+          wait = [body["retry_after"].to_f, config.search_retry_floor.to_f].max
+          log(:info) { "search index not ready for #{guild_id}; retrying in #{wait}s" }
+          sleep(wait)
+        end
+      end
+
+      # --- Guilds ------------------------------------------------------------
+
       # @return [Hash]
       def get_guild(guild_id)
         request(:get, "/guilds/#{guild_id}", route: "GET /guilds/#{guild_id}")
@@ -340,6 +402,40 @@ module DiscordStore
         rand * ceiling
       end
 
+      def search_request(guild_id, params, route)
+        request(:get, "/guilds/#{guild_id}/messages/search",
+                query: flatten_query(params), route: route)
+      rescue AuthError => e
+        # A 403 here is almost always the privileged intent rather than the
+        # token, and the generic message sends people to check credentials that
+        # are fine.
+        raise MissingIntentError if e.status == 403
+
+        raise
+      end
+
+      def index_pending?(body)
+        body.is_a?(Hash) && body["code"] == INDEX_NOT_READY_CODE
+      end
+
+      def normalise_search(body)
+        # messages is an array of arrays: it used to carry the surrounding
+        # context of each hit, and Discord kept the shape after dropping the
+        # context.
+        hits = Array(body["messages"]).flatten.compact
+        {
+          "messages" => hits.select { |message| own_message?(message) },
+          "total_results" => body["total_results"].to_i,
+          "indexing" => body["doing_deep_historical_index"] ? true : false
+        }
+      end
+
+      # Discord takes repeated keys for array params (author_id=1&author_id=2),
+      # which is what URI.encode_www_form does with an array value anyway.
+      def flatten_query(params)
+        params.reject { |_, value| value.nil? || (value.respond_to?(:empty?) && value.empty?) }
+      end
+
       def build_url(path, query)
         url = "#{config.api_base}#{path}"
         return url if query.nil? || query.empty?

data/lib/discord_store/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module DiscordStore
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/discord_store.rb

@@ -18,6 +18,9 @@ require_relative "discord_store/transport/rate_limiter"
 require_relative "discord_store/transport/rest"
 require_relative "discord_store/transport/fake"
 
+# Depends on Transport::REST's constants at load time, so it comes after it.
+require_relative "discord_store/search"
+
 # Uses Discord as a database.
 #
 # It works, in the sense that the data goes in and comes back out. It is also an
@@ -95,6 +98,17 @@ module DiscordStore
       @kv ||= KV.new(rest: @rest, config: @config)
     end
 
+    # Discord's message index.
+    #
+    # Not the read path -- see {Search} for why at length -- and needs the
+    # MESSAGE_CONTENT privileged intent. Exists for the queries that a channel
+    # scan answers badly, such as finding chunks whose manifest is gone.
+    #
+    # @return [Search]
+    def search
+      @search ||= Search.new(rest: @rest, config: @config)
+    end
+
     # @return [GuildLimits]
     def limits
       @limits ||= GuildLimits.new(rest: @rest, config: @config)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: discord_store
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Chayut
@@ -56,6 +56,7 @@ files:
 - lib/discord_store/log/record.rb
 - lib/discord_store/railtie.rb
 - lib/discord_store/replay.rb
+- lib/discord_store/search.rb
 - lib/discord_store/snowflake.rb
 - lib/discord_store/tasks.rake
 - lib/discord_store/transport/bucket.rb