parse-stack-next 5.7.6 → 5.8.0

data/README.md

@@ -4,6 +4,16 @@
 
 A full-featured Ruby client SDK for [Parse Server](http://parseplatform.org/). [parse-stack-next](https://github.com/neurosynq/parse-stack-next) is a Ruby client SDK, REST client, and Active Model ORM for [Parse Server](http://parseplatform.org/), combining a low-level API client, a query engine, an object-relational mapper (ORM), and a Cloud Code Webhooks rack application in a single gem.
 
+## What's new in 5.8
+
+- **5.8.0: MCP deployments can expose less than their users can read.** `Parse::Agent.new(fields: { Customer => %i[display_name timezone] })` narrows a class's `agent_fields` for one agent, so a user-facing assistant and an analytics endpoint in one process expose different subsets of the same model. It applies to queries, includes, aggregation, search, schema output, exports, and `semantic_search` text, and `query_class`/`count_objects`/`export_data` now refuse filtering or sorting on a hidden field. See [CHANGELOG.md](./CHANGELOG.md)
+- **5.8.0: Supported deployment patterns.** `MCPRackApp.user_scoped` serves signed-in application users (no master-key fallback, server-pinned identity and tenant), and `MCPRackApp.master_analytics` serves a shared read-only analytics endpoint that requires an operator `principal_resolver`. Cancellations and approval replies are now bound to the session's owner, revocation intervals are documented, and orphaned subscriptions are reaped. See [CHANGELOG.md](./CHANGELOG.md)
+- **5.8.0: Retrieval profiles for `semantic_search`.** Profiles you register on the server (for example `fast`, `balanced`, `precise`; none ship by default) compose hybrid search and reranking with budgets, spend accounting, and observable fallback; each call emits a sanitized `parse.retrieval.search` event, and `Parse::Retrieval::Benchmark` measures profiles on labeled cases. See [CHANGELOG.md](./CHANGELOG.md)
+- **5.8.0: Vector index definitions from the model, with optional quantization.** `Parse::VectorSearch::IndexDefinition` generates the Atlas definition from your declarations (filters and tenant field included) with preview and diff, and `quantization: :scalar`/`:binary` on a `:vector` property shrinks index memory without changing stored data. Contextualized embedding batches now adapt to provider size limits. See [CHANGELOG.md](./CHANGELOG.md)
+- **5.8.0: Two silent mismatches fixed; check whether you need to act.** Queries now send a property's declared `field:` name exactly as declared (`property :account_id, :string, field: :account_id` queries `account_id`, no longer `accountId`), so queries that matched nothing against underscore or mixed-case columns now return rows. Vector search, index auto-discovery, and drift checks now use the stored column of a multi-word `:vector` property: an Atlas vector index whose path is the Ruby name (`body_embedding`) must be recreated on the stored column (`bodyEmbedding`). Drift detection reports the mismatch. See [CHANGELOG.md](./CHANGELOG.md)
+
+See [CHANGELOG.md](./CHANGELOG.md) for the full 5.8 entry.
+
 ## What's new in 5.7
 
 - **5.7.6: `semantic_search` respects `agent_fields` for chunk content.** A class that embedded a field hidden by its `agent_fields` allowlist returned that field's text as chunk content. Chunk text now comes only from embedded fields the agent may read; a hidden text source is refused with `:field_denied` before the search runs. See [CHANGELOG.md](./CHANGELOG.md)
@@ -92,7 +102,7 @@ See [CHANGELOG.md](./CHANGELOG.md) for the full 5.1 entry, including breaking ch
 - **MCP transport hardening** — Streamable HTTP `Mcp-Session-Id` header (renamed from `X-MCP-Session-Id`, **breaking**), `MCP-Protocol-Version` validation, `DELETE /` session termination, structured-content (`outputSchema`) on built-in tools, optional `health_path:` liveness probe
 - **`Parse::GraphQL::TypeGenerator`** — generate `graphql-ruby` types directly from your `Parse::Object` subclasses (no Parse Server round-trip), with `:vector` columns surfaced as `[Float]` and association registries (`has_one_associations`, `has_many_associations`) populated at DSL time
 - **LiveQuery promoted to stable** — the experimental warning is removed; `Parse.live_query_enabled = true` is retained as a network-egress safety toggle, not a stability gate
-- **Server-version deprecation warning** — one-shot warning when connecting to a Parse Server older than the configured threshold (default `7.0.0`, override with `PARSE_DEPRECATED_SERVER_VERSION_BELOW`); silence with `Parse.suppress_server_version_warning = true`. The **supported baseline is Parse Server 9.x** (the SDK is developed and tested against a pinned `parse-server:9.10.0`); the default warning threshold is intentionally conservative so older deployments only get an advisory, not a hard break.
+- **Server-version deprecation warning**: a one-shot warning when connecting to a Parse Server older than the configured threshold (default `7.0.0`, override with `PARSE_DEPRECATED_SERVER_VERSION_BELOW`); silence with `Parse.suppress_server_version_warning = true`. The **supported baseline is Parse Server 9.x** (the SDK is developed and tested against a pinned `parse-server:9.10.3`); the default warning threshold is intentionally conservative so older deployments only get an advisory, not a hard break.
 - **`mongo_relation_index :field, dedup: true`** — register a compound `{owningId, relatedId}` UNIQUE on relation join collections to prevent duplicate-pair subscriptions without breaking `has_many` semantics
 
 See [CHANGELOG.md](./CHANGELOG.md) for the full 5.0 entry, including security-hardening notes and Ruby 3.x cleanup.
@@ -916,8 +926,8 @@ view = Parse.client.sdk_cache
 store.verify_upstream_isolation!
 ```
 
-**The two URLs must address different Redis databases.** On Parse Server 9.10.0
-and earlier, a `_Role` write clears the cache with `FLUSHDB`, which on a shared
+**The two URLs must address different Redis databases.** Through at least Parse
+Server 9.10.3, a `_Role` write clears the cache with `FLUSHDB`, which on a shared
 database deletes the SDK's cached responses and its create-locks along with it
 ([parse-server#10617](https://github.com/parse-community/parse-server/issues/10617)).
 `verify_upstream_isolation!` detects this by scanning the SDK's own database
@@ -6473,7 +6483,7 @@ export PARSE_TEST_SERVER_URL=http://localhost:29337/parse
 export PARSE_TEST_APP_ID=psnextItAppId
 export PARSE_TEST_API_KEY=psnext-it-rest-key
 export PARSE_TEST_MASTER_KEY=psnextItMasterKey
-export PARSE_TEST_MONGO_URI="mongodb://admin:password@localhost:29017/parse_stack_next_it?authSource=admin"
+export PARSE_TEST_MONGO_URI="mongodb://admin:password@localhost:29017/parse_stack_next_it?authSource=admin&directConnection=true"
 export PARSE_TEST_REDIS_URL=redis://localhost:29379/0
 export PARSE_TEST_LIVE_QUERY_URL=ws://localhost:29337
 export ATLAS_URI="mongodb://localhost:29020/parse_atlas_test?directConnection=true"

data/docs/TEST_SERVER.md

@@ -62,8 +62,8 @@ default.
 - **`PSNEXT_PREFIX`** (default `psnext-it`) names the Compose project and every
   container. Set it (e.g. `PSNEXT_PREFIX=psnext-ci`) to run a second, fully
   separate copy.
-- **Versions**: Parse Server is pinned to `parseplatform/parse-server:9.9.0`
-  (see `scripts/docker/Dockerfile.parse`), MongoDB `mongo:8`, Redis
+- **Versions**: Parse Server is pinned to `parseplatform/parse-server:9.10.3`
+  (see `scripts/docker/Dockerfile.parse`), MongoDB `mongo:9` (`MONGO_VERSION`), Redis
   `redis:7-alpine`, Dashboard `parseplatform/parse-dashboard:9`.
 - **Database**: Parse uses `parse_stack_next_it`.
 

data/docs/acl_clp_guide.md

@@ -491,6 +491,13 @@ described in §7. From a security standpoint, an Atlas Search call
 with a session token is treated like a `Parse::Query` with a session
 token — same scoping, same field stripping.
 
+Stripping a protected field from results does not stop it from
+deciding which documents match. A scoped `Parse::AtlasSearch.search`
+is therefore refused (`Parse::CLPScope::Denied`) when `fields:` names a
+field protected for the caller, or when it names no fields (a search
+over every column) while the caller has any protected fields. Pass
+`fields:` with the fields to search.
+
 The `$search` stage itself runs on the Atlas Search index and is not
 filtered by ACL. The ACL filter is applied as a `$match` stage by
 `Parse::ACLScope` after `$search`, before results are returned. If

data/docs/atlas_vector_search_guide.md

@@ -209,7 +209,7 @@ Parse::AtlasSearch::IndexCatalog.create_index(
     fields: [
       {
         type: "vector",
-        path: "body_embedding",
+        path: "bodyEmbedding",          # the STORED column, not the Ruby name
         numDimensions: 1536,
         similarity: "cosine",
       },
@@ -221,6 +221,12 @@ Parse::AtlasSearch::IndexCatalog.create_index(
 )
 ```
 
+The vector `path` is the column the property is stored under:
+`property :body_embedding, :vector` is saved as `bodyEmbedding` (or under an
+explicit `field:` alias). `find_similar`, hybrid search, index discovery, and
+drift checks all use that stored name; `Parse::VectorSearch::IndexDefinition`
+generates it for you.
+
 Including `_rperm` as a filter field lets the per-row ACL match
 short-circuit at the index level — strongly recommended for any
 field that ACL-scoped agents will search against.
@@ -241,6 +247,91 @@ indexes for one whose definition covers the requested `path`. The
 first match wins; pass `index:` explicitly when you have more than
 one covering index and want a specific one.
 
+### Generating the definition from the model
+
+Writing the definition by hand lets it drift from the model: a changed
+`dimensions:`, a new `agent_searchable` filter field, or a tenant scope
+the index does not cover. `Parse::VectorSearch::IndexDefinition` derives
+the definition from the model's own declarations instead:
+
+* the `:vector` property's path, `dimensions:`, `similarity:` (`cosine`
+  when undeclared), and optional `quantization:`;
+* every `agent_searchable filter_fields:` entry as a `type: "filter"` path
+  (pointer fields use their `_p_<column>` storage path);
+* the `agent_tenant_scope` field, which retrieval folds into
+  `$vectorSearch.filter` on every scoped query.
+
+Output is deterministic (vector entry first, filters sorted by path), so
+it diffs cleanly in review.
+
+```ruby
+class Document < Parse::Object
+  property :category, :string
+  property :embedding, :vector, dimensions: 1024, similarity: :dotProduct
+  agent_searchable field: :embedding, filter_fields: %i[category]
+
+  # Declare the index; its definition is generated, not written by hand.
+  vector_search_index "document_vec"
+end
+
+# Preview without touching Atlas.
+Parse::VectorSearch::IndexDefinition.build(Document)
+Parse::Schema.vector_index_definition(Document)          # same thing
+Parse::VectorSearch::IndexDefinition.preview(Document, name: "document_vec")
+
+# Compare against what is deployed (a definition or a $listSearchIndexes entry).
+live = Parse::AtlasSearch::IndexCatalog.find_vector_index("Document", field: :embedding)
+Parse::VectorSearch::IndexDefinition.diff(Parse::Schema.vector_index_definition(Document), live)
+# => { in_sync: false,
+#      vector: { "numDimensions" => { declared: 1024, live: 1536 } },
+#      filters_missing: ["category"], filters_extra: [] }
+```
+
+Applying stays explicit. `vector_search_index` declarations join the
+model's `mongo_search_index` declarations in
+`Parse::Schema::SearchIndexMigrator`, which plans first and only mutates
+Atlas when asked; a drifted index is reported, not rebuilt, unless you
+pass `update: true`. The definition is generated when the migrator plans,
+so `agent_searchable` and `agent_tenant_scope` can be declared in any
+order.
+
+```ruby
+Document.search_indexes_plan          # :to_create / :in_sync / :drifted / :orphans
+Document.apply_search_indexes!        # creates missing indexes only
+Document.apply_search_indexes!(update: true)   # also rebuilds drifted ones
+```
+
+### Index-side quantization
+
+Atlas can quantize a float vector field when it builds the index, which
+shrinks the index Atlas keeps in memory. Declare it per property; it is
+off by default:
+
+```ruby
+property :embedding, :vector, dimensions: 1024, quantization: :scalar   # or :binary
+```
+
+The generated definition then carries `"quantization": "scalar"` (or
+`"binary"`) on the vector field. Only the index changes: stored vectors and
+the SDK write path stay full precision, so turning it on or off is an index
+rebuild, not a re-embed.
+
+| Setting | Index memory | Recall |
+|---------|--------------|--------|
+| none (default) | full float vectors | baseline |
+| `:scalar` | roughly 4x smaller | small loss for most embedding models |
+| `:binary` | roughly 32x smaller | larger loss; Atlas rescoring recovers part of it |
+
+Measure recall on your own queries before adopting `:binary`; the right
+choice depends on the embedding model and the collection. For small
+collections the memory saving rarely matters.
+
+First-query drift verification also checks quantization: an index whose
+`quantization` differs from the property's declaration (an absent value
+counts as none on either side) is reported under
+`Parse::VectorSearch.index_drift_policy` like a dimension or similarity
+mismatch.
+
 ---
 
 ## Running similarity queries: `find_similar`
@@ -302,6 +393,9 @@ index's `latestDefinition` against the model declaration:
   (usually an index that predates a model change).
 * `similarity` vs the property's declared `similarity:` (checked only
   when both sides declare one).
+* `quantization` vs the property's declared `quantization:` (absent on
+  either side means none, so an index quantized without a declaration
+  is drift too).
 * When the class registers an `agent_tenant_scope`, the scope field
   must appear among the index's `type: "filter"` paths — without it,
   every tenant-scoped `$vectorSearch.filter` fails Atlas-side at
@@ -505,18 +599,27 @@ Two things to know:
   one-chunk document. Registering a context model as an `embed` provider
   works, but the stored vectors carry no surrounding-document context.
   Use `embed_chunks` when chunk-level context is the point.
-* **Request sizing is partly handled for you.** Voyage caps one request
-  at 1,000 documents, 16,000 chunks, and 120k tokens. `embed_chunks`
-  checks the document and chunk caps before sending, and splits large
-  inputs across several requests by whole document (a document's chunks
-  always travel together) so that each **response** stays under the
-  SDK's response-size cap. That split is sized by the vectors coming
-  back, not by tokens going out, so it does **not** guarantee a request
-  stays under the 120k input-token cap. The SDK has no tokenizer to
-  check that; keep long documents to a few per call, and expect a
-  `BadRequestError` from Voyage if a request exceeds it. Context models
-  default to `embed_batch_size: 32` to keep `embed_text` batches clear of
-  the token cap for typical inputs, which is a heuristic, not a check.
+* **Request sizing adapts; it is not exact token counting.** Voyage caps
+  one request at 1,000 documents, 16,000 chunks, and 120k input tokens.
+  `embed_chunks` (and `embed_text` on a context model) packs whole
+  documents into requests that stay within the document cap, a chunk
+  count whose **response** fits the SDK's response-size cap, and an
+  **estimated** 120k-token budget. The SDK has no tokenizer, so the
+  estimate assumes three bytes per token, which over-counts typical
+  English text and packs conservatively. A document's chunks always
+  travel together.
+  When Voyage still rejects a request as too large (its "batch size" or
+  "max allowed tokens per submitted batch" errors, or an HTTP 413), the
+  SDK halves that request by document and resends each half, keeping
+  the returned vectors aligned with your input. Only those positively
+  identified size errors trigger a split; any other 400 is raised as is,
+  with the provider's message on `BadRequestError#detail`. A document
+  that is rejected as too large even on its own raises a
+  `BadRequestError` naming its index, since no split can fix it: break
+  it into fewer or shorter chunks. A single chunk longer than the
+  model's context window ("tokens in an example exceeds the context
+  length") is also raised directly. Context models default to
+  `embed_batch_size: 32` for `embed_text` batches.
 
 ---
 
@@ -678,6 +781,71 @@ adapters implement only the network call (`#rerank_scores`).
 > rate-limited tool error). Admin agents are exempt; direct
 > `find_similar` / `retrieve` callers are not metered.
 
+### Retrieval profiles for `semantic_search` (5.8)
+
+An agent can choose among a few server-configured retrieval strategies by
+name, without ever choosing a provider, endpoint, or credential. Register
+rerankers by name, then profiles that reference them:
+
+```ruby
+Parse::Retrieval.register_reranker(:voyage,
+  Parse::Retrieval::Reranker::Voyage.new(api_key: ENV.fetch("VOYAGE_API_KEY"), model: "rerank-3-lite"))
+
+Parse::Retrieval::Profiles.register(:fast, k: 5, max_k: 10)
+Parse::Retrieval::Profiles.register(:balanced, k: 8, hybrid: true)
+Parse::Retrieval::Profiles.register(:precise,
+  k: 8, reranker: :voyage,
+  rerank_candidates: 30,            # documents retrieved and sent to the reranker (max 100)
+  rerank_top_n: 8,                  # documents kept after reranking
+  rerank_max_document_chars: 4_000, # each document's text is cut before it leaves the process
+  rerank_timeout: 5,                # seconds
+  on_rerank_failure: :fallback,     # or :raise
+  max_total_tokens: 8_000)          # default response budget
+
+agent.execute(:semantic_search, class_name: "Article", query: "refund policy", profile: "precise")
+```
+
+* **Validated at registration.** An unknown option, an unregistered reranker,
+  `k` above `max_k`, or a non-positive budget raises `ArgumentError` at boot.
+  An unknown profile name at call time is refused with the list of available
+  profiles.
+* **Field-safe reranking.** The reranker receives the same text source as
+  chunk content, which `semantic_search` restricts to fields the agent may
+  read (its effective `agent_fields`, including any per-agent `fields:`
+  narrowing), cut to `rerank_max_document_chars`.
+* **Spend.** Estimated rerank tokens are charged to the same per-tenant
+  `SpendCap` budget as the query embedding (admin agents are exempt).
+* **Fallback is observable.** On a reranker timeout or provider error,
+  `:fallback` keeps the retrieval order and adds `rerank_fallback: true` and
+  `rerank_fallback_reason` to the result; `:raise` fails the call.
+* **Defaults are unchanged.** Without `profile:` the tool behaves as before.
+
+Each call emits one `parse.retrieval.search` notification with the profile,
+`k`, candidate count, rerank stats (`used`, `documents`, `chars`,
+`tokens_estimated`, `duration_ms`, `fallback`, `fallback_reason`), chunk and
+document counts, budget drops, and timings. It never carries the query,
+document text, field values, URLs, or credentials, and `tokens_estimated` is
+the SDK's estimate, not provider-reported usage.
+
+**Measuring profiles.** `Parse::Retrieval::Benchmark` scores profiles on a
+labeled case set (recall@k, MRR, hit rate, mean and p95 latency, estimated
+rerank tokens), overall and per tag. Cases may list `forbidden` ids that must
+never be returned (restrictive ACLs, other tenants); any such hit is reported
+as a violation.
+
+```ruby
+cases = Parse::Retrieval::Benchmark.load_cases("eval/cases.json")
+runner = Parse::Retrieval::Benchmark.semantic_search_runner(agent, class_name: "Article")
+# nil is the baseline (no profile); its results are reported under "default".
+report = Parse::Retrieval::Benchmark.run(cases: cases, profiles: [nil, "fast", "precise"], runner: runner)
+report["default"]                          # => the baseline, for comparison
+report["precise"]                          # => { recall_at_k:, mrr:, hit_rate:, violations:, mean_ms:, p95_ms:, ... }
+report["precise"][:by_tag]["long_document"]
+```
+
+Run it against your own data and Atlas index before recommending a profile;
+the SDK does not ship measured defaults.
+
 ### Chunkers
 
 The default is a fixed-size sliding window with overlap. Subclass

data/docs/client_sdk_guide.md

@@ -1176,6 +1176,17 @@ server-side on every event before it goes out the WebSocket — Bob will
 not receive an event for an ACL-private row Alice creates, even if his
 subscription matches the `where` clause.
 
+> **Role-based CLP grants need a server option.** Parse Server does not
+> resolve the subscriber's roles when LiveQuery evaluates Class Level
+> Permissions unless `enableLiveQueryClassLevelPermissionRoles: true` is
+> set (env `PARSE_SERVER_ENABLE_LIVE_QUERY_CLASS_LEVEL_PERMISSION_ROLES`,
+> available from Parse Server 9.10.3, default `false`). With the default,
+> a CLP such as `"find" => { "role:Editor" => true }` admits role members
+> over REST but rejects their LiveQuery subscription. Deployments that
+> grant LiveQuery access through `role:` entries in CLP must turn it on.
+> Object ACL `role:` entries are honored either way. The SDK test stack
+> enables it in `scripts/start-parse.sh`.
+
 > **Master-key authorization is per-CONNECTION, not per-subscription.**
 > Parse Server resolves master-key (ACL/CLP-bypass) authorization once,
 > from the connect frame; once set, EVERY subscription on that socket