exwiw 0.9.1 → 0.9.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 577420a290ea24adc64ef6ba21914ac71b1c98f86163f9d1d8a88643cc89e15b
-  data.tar.gz: eaddb0073169543aa1bea944cb1b075fbd8258a034f4cffb9cf0011338ba260a
+  metadata.gz: 1856ad12991806b66ecce4a9fa6d85bc89c9e371f38c90da14168367b99e4f5b
+  data.tar.gz: 0fe953311f016847e4b4adf27c8b82ff94d3012d8d896406200b4d94080e442e
 SHA512:
-  metadata.gz: c21735b0dcda7b30b0519944c37f24610564e91801a29e2c0f7cc3632cc0f24c532d10c078127bccffd0a2f258a5695cdabe25094edf228b5e3203a629ddf175
-  data.tar.gz: 7e9ce9682b281c097485da409befc466dd06fc655c1612278b9e98f6bccdea575aa443ff4292777be1e4e68f88d826554702eecd6b94bddc31b1e4edc01c923b
+  metadata.gz: bb984601846145d7e542bdaddfecfb836a393926f3cd985f879bdabd3a5c28a37b190de443ae881faf2adec384429746dd1816b2d5c5ad0d11d4f570dbc94e80
+  data.tar.gz: 492316c915507618455644f2f4ff2073479db979a6e18b70aa8e2ff1efeb4df76562770be3bf760b3ab628c863ac3bb3b5a70806bc029730bda29df0bdb90ec2

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## [Unreleased]
 
+## [0.9.2] - 2026-06-30
+
+### Added
+
+- **The mongodb adapter accepts a server-enforced query timeout, globally and per collection.** A global `--mongodb-query-timeout-ms=N` (also `mongodb_query_timeout_ms:` in the config file) sets the MongoDB driver's CSOT `timeout_ms` on the client, so every operation — the find cursor's whole lifetime (initial batch and every `getMore` the streaming dump walks), the document count, and an executing `explain` — is bounded; past the deadline the server aborts the operation and exwiw fails the run, so an accidentally heavy or unscoped query cannot keep pinning the (often production) source. A per-collection `query_timeout_ms` key in the schema config overrides the global for that collection's find/count (give a known-large collection more headroom, or cap a runaway one); like `bulk_insert_chunk_size`, it is user-maintained and preserved across Mongoid schema regeneration. Both are **mongodb-only** (the SQL adapters shell out to their own clients) and must be positive integers; the default is no timeout. The fork-parallel dump inherits both automatically (workers rebuild the client from the same connection config and run the same per-collection queries).
+
 ## [0.9.1] - 2026-06-30
 
 ### Fixed

data/README.md

@@ -300,6 +300,7 @@ insert_only: false
 after_insert_hook: hooks/seed.rb
 log_level: info              # debug | info
 # target_table / ids / ids_field / scope_column may also be set here
+# mongodb_query_timeout_ms: 30000   # global query timeout (mongodb only)
 ```
 
 With the file above, only the connection details need to be supplied on the CLI:
@@ -317,6 +318,7 @@ Notes:
 - Unknown keys are rejected so a typo surfaces immediately.
 - Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain`, so a single config file can be shared by both subcommands.
 - `explain_verbosity` sets the mongodb `explain` verbosity (`queryPlanner` | `executionStats` | `allPlansExecution`, default `queryPlanner`); the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` env var overrides it. Ignored by the SQL adapters and by `export`. See [`exwiw explain`](#mongodb-explain-verbosity).
+- `mongodb_query_timeout_ms` sets the global, server-enforced query timeout (mongodb only); the `--mongodb-query-timeout-ms` CLI flag overrides it. Ignored by the SQL adapters. See [MongoDB notes](#mongodb-notes).
 
 ### Generator
 
@@ -395,7 +397,7 @@ It is a distinct task and class (`Exwiw::MongoidSchemaGenerator`) from the Activ
 
 Models in an inheritance hierarchy whose subclasses share the base's collection (Mongoid STI, distinguished by the auto-added `_type` discriminator) collapse into a single config: the generator discovers the subclasses via `descendants` (Mongoid registers only the base class in `Mongoid.models`) and unions every class's `fields` and `belongs_tos` into the collection config, so subclass-only fields and associations are not lost.
 
-Regeneration preserves hand-edited `replace_with`, `filter`, `ignore`, and `bulk_insert_chunk_size` values, like the ActiveRecord generator. Indexes are not written to the config — they are introspected from the live database at dump time (see [MongoDB notes](#mongodb-notes)). Polymorphic `belongs_to` is not yet expanded by this task.
+Regeneration preserves hand-edited `replace_with`, `filter`, `ignore`, `bulk_insert_chunk_size`, and `query_timeout_ms` values, like the ActiveRecord generator. Indexes are not written to the config — they are introspected from the live database at dump time (see [MongoDB notes](#mongodb-notes)). Polymorphic `belongs_to` is not yet expanded by this task.
 
 By default the task **aborts** when a model uses a construct exwiw cannot represent: a `belongs_to` whose target class can no longer be resolved (a stale relation left behind after its model was removed), or a polymorphic / self-referential-cyclic / ambiguous / unresolvable-parent `embedded_in` (see the cases above).
 
@@ -800,6 +802,7 @@ The MongoDB adapter is experimental. To use it:
 - `--target-collection=COLLECTION` is a mongodb-only alias of `--target-table` (use whichever reads better for MongoDB). Specifying both, or using `--target-collection` with a non-mongodb adapter, is an error.
 - `--ids-field=FIELD` matches `--ids` against `FIELD` on the target collection instead of its primary key (e.g. `--target-collection=users --ids=a@example.com --ids-field=email`). Downstream foreign-key propagation still keys off the primary key, so only the target collection's filter changes. Unlike the primary-key path, the supplied ids are **not** type-coerced (the stored type of a custom field is unknown), so pass values matching the field's actual type. This flag is **mongodb-only** (the SQL adapters have no equivalent).
 - Large or embedded-document-heavy dumps are streamed automatically: the adapter reads the collection through a lazy cursor (not `.to_a`) and writes JSONL in chunks, so peak memory is bounded by the chunk size rather than the collection size — no flag to set. Encoding each document to MongoDB Extended JSON is accelerated by an **optional native (C) extension** that compiles automatically on `gem install`; where it cannot compile, exwiw falls back to a byte-identical pure-Ruby encoder. See [`docs/optimization-notes.md`](docs/optimization-notes.md) for the performance investigation and [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md) for the native encoder's design. Benchmark your own data with `script/bench_mongodb_dump.rb`.
+- `--mongodb-query-timeout-ms=N` sets a global, **server-enforced** timeout (in milliseconds) on every query exwiw issues — the find cursor's whole lifetime (the initial batch and every `getMore` the streaming dump walks), the count, and an executing `explain`. Past the deadline the server aborts the operation and exwiw fails the run, so an accidentally heavy or unscoped query cannot keep pinning the (often production) source. It is **mongodb-only** (the SQL adapters shell out to their own clients) and may also be set as `mongodb_query_timeout_ms:` in the config file. The default is no timeout. A single collection can opt to a different limit with a `query_timeout_ms` key in its schema config (a sibling of `bulk_insert_chunk_size`), which overrides the global for that collection's find/count — use it to give a known-large collection more headroom, or to cap one that tends to run away. Hand-edited `query_timeout_ms` values are preserved across schema regeneration.
 - `--parallel-workers=N` (opt-in, `export` only) forks `N` worker processes that decode whole collections in parallel — the dominant cost on a large dump is the driver's BSON→Ruby decode, and each worker decodes its own collections in their natural order, so the output stays **byte-identical** to a serial run (same filenames and content). It needs a dump target (the schedule is built around the scoped DAG) and a `fork`-capable runtime (CRuby on POSIX), falling back to the serial path otherwise; it also accepts `parallel_workers:` in the config file. The speedup needs real cores to spend — it reaches ~2× from 4 workers and saturates there. The default is serial. See [`docs/mongodb-dump-parallelism-2x-notes.md`](docs/mongodb-dump-parallelism-2x-notes.md) for the schedule and measurements.
 - Output is JSON Lines (`insert-{idx}-{collection}.jsonl`) using MongoDB Extended JSON (relaxed mode). Import with `mongoimport`:
   ```bash

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -40,15 +40,20 @@ module Exwiw
       class StreamingResult
         include Enumerable
 
-        def initialize(view:, collection:, keys:, state:)
+        def initialize(view:, collection:, keys:, state:, timeout_ms: nil)
           @view = view
           @collection = collection
           @keys = keys
           @state = state
+          @timeout_ms = timeout_ms
         end
 
         def size
-          @size ||= @view.count_documents
+          # count_documents reads :timeout_ms only from the opts passed here (it
+          # does not inherit the find view's per-op timeout), so the per-collection
+          # value must be threaded in explicitly. When nil it falls back to the
+          # client-wide global timeout, like every other operation.
+          @size ||= @timeout_ms ? @view.count_documents(timeout_ms: @timeout_ms) : @view.count_documents
         end
         alias length size
 
@@ -172,6 +177,7 @@ module Exwiw
           primary_key: config.primary_key,
           filter: filter,
           projection: build_projection(config, @propagation_keys),
+          timeout_ms: config.query_timeout_ms,
         )
       end
 
@@ -179,7 +185,7 @@ module Exwiw
         @logger.debug("  Executing Mongo find on '#{query.collection}': filter=#{query.filter.inspect} projection=#{query.projection.inspect}")
 
         view = db[query.collection]
-          .find(query.filter)
+          .find(query.filter, find_timeout_opts(query))
           .projection(query.projection)
           .comment(query_comment_text("collection=#{query.collection}"))
 
@@ -195,7 +201,7 @@ module Exwiw
         # large / embed-heavy collections — the dump's dominant memory cost. The
         # propagation-key values are captured as the cursor streams and published
         # into @state once the pass completes (see StreamingResult).
-        StreamingResult.new(view: view, collection: query.collection, keys: keys, state: @state)
+        StreamingResult.new(view: view, collection: query.collection, keys: keys, state: @state, timeout_ms: query.timeout_ms)
       end
 
       # NOTE: relies on @embedded_children_by_parent set by a prior build_query
@@ -233,7 +239,7 @@ module Exwiw
         @logger.debug("  Running explain (verbosity=#{verbosity}) on '#{query.collection}': filter=#{query.filter.inspect}")
 
         result = db[query.collection]
-          .find(query.filter)
+          .find(query.filter, find_timeout_opts(query))
           .projection(query.projection)
           .comment(query_comment_text("collection=#{query.collection}"))
           .explain(verbosity: verbosity)
@@ -360,6 +366,16 @@ module Exwiw
         false
       end
 
+      # Per-operation find options carrying the collection's CSOT timeout. An
+      # empty hash when the query has none, so the operation inherits the
+      # client-wide global timeout (or runs untimed if that is also unset). The
+      # find view's :timeout_ms governs the whole cursor lifetime — initial batch
+      # plus every getMore the streaming dump walks — which is what makes it the
+      # right cap for an accidentally heavy/unscoped extraction.
+      private def find_timeout_opts(query)
+        query.timeout_ms ? { timeout_ms: query.timeout_ms } : {}
+      end
+
       private def reject_filter!(config)
         return if config.filter.nil? || config.filter.to_s.empty?
 
@@ -670,14 +686,14 @@ module Exwiw
               # given, overrides the database in the URI path; otherwise the
               # URI's own database is used. The URI is never logged (it may carry
               # credentials).
-              client_options = {}
+              client_options = global_timeout_options
               if @connection_config.database_name && !@connection_config.database_name.to_s.empty?
                 client_options[:database] = @connection_config.database_name
               end
               Mongo::Client.new(@connection_config.uri, **client_options)
             else
               address = "#{@connection_config.host}:#{@connection_config.port}"
-              options = { database: @connection_config.database_name }
+              options = global_timeout_options.merge(database: @connection_config.database_name)
               if @connection_config.user && !@connection_config.user.to_s.empty?
                 options[:user] = @connection_config.user
                 options[:password] = @connection_config.password
@@ -690,6 +706,15 @@ module Exwiw
       private def uri_connection?
         !@connection_config.uri.nil? && !@connection_config.uri.to_s.empty?
       end
+
+      # Client-level CSOT default applied to every operation on this connection
+      # (find cursor lifetime, count, executing explain). nil when no global
+      # timeout is configured, leaving the client untimed; a per-collection
+      # `query_timeout_ms` still overrides this per find/count.
+      private def global_timeout_options
+        timeout = @connection_config.mongodb_query_timeout_ms
+        timeout ? { timeout_ms: timeout } : {}
+      end
     end
   end
 end

data/lib/exwiw/cli.rb

@@ -37,6 +37,7 @@ module Exwiw
       scope_column
       parallel_workers
       explain_verbosity
+      mongodb_query_timeout_ms
     ].freeze
 
     # MongoDB explain verbosity levels (passed through to the server's explain
@@ -89,6 +90,7 @@ module Exwiw
       @insert_only = nil
       @after_insert_hook_path = nil
       @parallel_workers = nil
+      @mongodb_query_timeout_ms = nil
       @explain_verbosity = nil
       # nil (not :info) so we can tell "user passed --log-level" from the default,
       # letting a config-file value fill in; the :info default is applied later.
@@ -113,6 +115,7 @@ module Exwiw
         password: @database_password,
         database_name: @database_name,
         uri: @connection_uri,
+        mongodb_query_timeout_ms: @mongodb_query_timeout_ms,
       )
 
       dump_target = DumpTarget.new(
@@ -178,6 +181,7 @@ module Exwiw
       resolve_ids_field!
       resolve_uri_option!
       resolve_parallel_workers!
+      resolve_mongodb_query_timeout_ms!
 
       if @subcommand == "explain"
         validate_explain_only!
@@ -331,6 +335,7 @@ module Exwiw
       @ids_field ||= config["ids_field"]
       @scope_column ||= config["scope_column"]
       @parallel_workers ||= parse_parallel_workers(config["parallel_workers"]) if config.key?("parallel_workers")
+      @mongodb_query_timeout_ms ||= parse_mongodb_query_timeout_ms(config["mongodb_query_timeout_ms"]) if config.key?("mongodb_query_timeout_ms")
       @explain_verbosity ||= config["explain_verbosity"]
     end
 
@@ -457,6 +462,38 @@ module Exwiw
       exit 1
     end
 
+    # `--mongodb-query-timeout-ms` sets the global, server-enforced CSOT timeout
+    # applied to every MongoDB query (find cursor lifetime, count, executing
+    # explain). It is mongodb-only (the SQL adapters shell out to their own
+    # clients) and must be a positive integer. A per-collection `query_timeout_ms`
+    # in the schema config overrides it. Runs after the adapter name is normalized
+    # so the family check is reliable.
+    private def resolve_mongodb_query_timeout_ms!
+      return if @mongodb_query_timeout_ms.nil?
+
+      if @database_adapter != "mongodb"
+        $stderr.puts "--mongodb-query-timeout-ms is only supported by the mongodb adapter"
+        exit 1
+      end
+
+      if @mongodb_query_timeout_ms < 1
+        $stderr.puts "--mongodb-query-timeout-ms must be a positive integer (got #{@mongodb_query_timeout_ms})"
+        exit 1
+      end
+    end
+
+    # Coerce a config-file `mongodb_query_timeout_ms` (YAML scalar) to Integer,
+    # matching the CLI flag's Integer coercion. A non-integer value is a config
+    # typo, so fail fast rather than silently dropping it.
+    private def parse_mongodb_query_timeout_ms(value)
+      return nil if value.nil?
+
+      Integer(value)
+    rescue ArgumentError, TypeError
+      $stderr.puts "config 'mongodb_query_timeout_ms' must be an integer (got #{value.inspect})"
+      exit 1
+    end
+
     private def validate_explain_only!
       rejected = []
       rejected << "--output-dir" unless @output_dir.nil?
@@ -591,6 +628,7 @@ module Exwiw
           @after_insert_hook_path = File.expand_path(v)
         end
         opts.on("--parallel-workers=N", Integer, "Fork N workers for the MongoDB dump's parallel schedule (mongodb + export only; N>=2 enables it, default is serial). Output is byte-identical to serial; falls back to serial where fork is unavailable.") { |v| @parallel_workers = v }
+        opts.on("--mongodb-query-timeout-ms=N", Integer, "Global server-enforced timeout (ms) for every MongoDB query (mongodb only). Aborts an accidentally heavy/unscoped query past the deadline. Overridden per collection by `query_timeout_ms` in the schema config.") { |v| @mongodb_query_timeout_ms = v }
         opts.on("--log-level=LEVEL", "Log level (debug, info). default is info") { |v| @log_level = v.to_sym }
 
         opts.on("--help", "Print this help") do

data/lib/exwiw/mongo_query.rb

@@ -2,14 +2,19 @@
 
 module Exwiw
   module MongoQuery
-    Find = Struct.new(:collection, :primary_key, :filter, :projection, keyword_init: true) do
+    # `timeout_ms` is the per-collection, server-enforced operation timeout (CSOT)
+    # applied when this query runs; nil falls back to the client-wide global
+    # (ConnectionConfig#mongodb_query_timeout_ms) or, if that is also unset, no
+    # timeout. Omitted from #to_h when nil so the historical 4-key query shape is
+    # unchanged for collections without a timeout.
+    Find = Struct.new(:collection, :primary_key, :filter, :projection, :timeout_ms, keyword_init: true) do
       def to_h
         {
           collection: collection,
           primary_key: primary_key,
           filter: filter,
           projection: projection,
-        }
+        }.tap { |h| h[:timeout_ms] = timeout_ms unless timeout_ms.nil? }
       end
     end
   end

data/lib/exwiw/mongodb_collection_config.rb

@@ -13,6 +13,13 @@ module Exwiw
     attribute :belongs_tos, array(BelongsTo)
     attribute :fields, array(MongodbField)
     attribute :bulk_insert_chunk_size, optional(Integer), skip_serializing_if_nil: true
+    # Per-collection server-enforced query timeout in milliseconds (CSOT). Applied
+    # to this collection's find (whole cursor lifetime), count, and an executing
+    # explain; overrides the global ConnectionConfig#mongodb_query_timeout_ms. Use
+    # it to give a known-large collection more headroom than the global default
+    # (or, conversely, to cap one that tends to run away). User-maintained — the
+    # generator never emits it, so #merge carries it forward across regeneration.
+    attribute :query_timeout_ms, optional(Integer), skip_serializing_if_nil: true
     attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
     # Free-form note. Purely informational — exwiw never reads it — and preserved
     # across `MongoidSchemaGenerator` regeneration like the field / belongs_to
@@ -62,7 +69,8 @@ module Exwiw
     # - structural facts come from the freshly generated config: primary_key,
     #   belongs_tos, embedded_in.
     # - user customizations are kept from the receiver: filter, ignore,
-    #   bulk_insert_chunk_size, and each field's `replace_with` masking rule.
+    #   bulk_insert_chunk_size, query_timeout_ms, and each field's `replace_with`
+    #   masking rule.
     # - generated fields drive the field list (so added/removed fields track the
     #   model), but a matching receiver field wins to retain its masking.
     def merge(passed)
@@ -73,6 +81,7 @@ module Exwiw
         merged.primary_key = passed.primary_key
         merged.filter = filter
         merged.bulk_insert_chunk_size = bulk_insert_chunk_size
+        merged.query_timeout_ms = query_timeout_ms
         merged.ignore = ignore
         merged.ignore_type = ignore_type
         # A freshly generated comment (e.g. the skip_unsupported marker) wins so

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.1"
+  VERSION = "0.9.2"
 end

data/lib/exwiw.rb

@@ -56,5 +56,13 @@ module Exwiw
   # mongodb adapter, e.g. `mongodb+srv://...`). When present it is the source of
   # truth for the connection — host/port/user/password are ignored — so TLS,
   # replica_set, auth_source, etc. can be expressed via the URI's query string.
-  ConnectionConfig = Struct.new(:adapter, :host, :port, :user, :password, :database_name, :uri, keyword_init: true)
+  #
+  # `mongodb_query_timeout_ms` is the global, server-enforced operation timeout
+  # (CSOT `timeout_ms`) applied to every MongoDB query exwiw issues — the find
+  # cursor's whole lifetime, the count, and an executing `explain`. It guards
+  # against an accidentally heavy/unscoped query pinning the (often production)
+  # source: the server aborts the operation past the deadline. nil leaves it
+  # unset (no timeout). A per-collection `query_timeout_ms` overrides it.
+  # mongodb adapter only.
+  ConnectionConfig = Struct.new(:adapter, :host, :port, :user, :password, :database_name, :uri, :mongodb_query_timeout_ms, keyword_init: true)
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.1
+  version: 0.9.2
 platform: ruby
 authors:
 - Shia