parse-stack-next 5.7.6 → 5.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +1148 -0
- data/README.md +39 -23
- data/docs/TEST_SERVER.md +2 -2
- data/docs/acl_clp_guide.md +7 -0
- data/docs/atlas_vector_search_guide.md +181 -13
- data/docs/caching.md +50 -0
- data/docs/client_sdk_guide.md +11 -0
- data/docs/mcp_guide.md +332 -9
- data/docs/mongodb_direct_guide.md +27 -0
- data/docs/usage_guide.md +38 -0
- data/docs/webhooks_guide.md +126 -17
- data/lib/parse/acl_scope.rb +230 -50
- data/lib/parse/agent/approval_gate.rb +0 -0
- data/lib/parse/agent/constraint_translator.rb +42 -15
- data/lib/parse/agent/describe.rb +11 -3
- data/lib/parse/agent/field_names.rb +53 -0
- data/lib/parse/agent/field_policy.rb +74 -0
- data/lib/parse/agent/mcp_deployments.rb +426 -0
- data/lib/parse/agent/mcp_dispatcher.rb +8 -1
- data/lib/parse/agent/mcp_rack_app.rb +424 -45
- data/lib/parse/agent/mcp_server.rb +23 -1
- data/lib/parse/agent/mcp_subscriptions.rb +124 -6
- data/lib/parse/agent/metadata_registry.rb +67 -8
- data/lib/parse/agent/prompt_hardening.rb +9 -3
- data/lib/parse/agent/tools.rb +378 -29
- data/lib/parse/agent.rb +298 -21
- data/lib/parse/api/batch.rb +10 -1
- data/lib/parse/api/schema.rb +23 -4
- data/lib/parse/api/sessions.rb +6 -2
- data/lib/parse/api/users.rb +88 -14
- data/lib/parse/atlas_search/index_manager.rb +5 -1
- data/lib/parse/atlas_search/protected_paths.rb +236 -0
- data/lib/parse/atlas_search.rb +144 -24
- data/lib/parse/authorization.rb +289 -4
- data/lib/parse/cache/sub_cache.rb +26 -0
- data/lib/parse/client/authentication.rb +19 -1
- data/lib/parse/client/batch.rb +264 -35
- data/lib/parse/client/body_builder.rb +21 -0
- data/lib/parse/client/caching.rb +371 -27
- data/lib/parse/client/request.rb +26 -14
- data/lib/parse/client/response.rb +49 -6
- data/lib/parse/client.rb +227 -40
- data/lib/parse/clp_scope.rb +317 -25
- data/lib/parse/console.rb +2 -2
- data/lib/parse/embeddings/voyage.rb +181 -17
- data/lib/parse/graphql/type_generator.rb +3 -0
- data/lib/parse/live_query/client.rb +72 -2
- data/lib/parse/lock_backend.rb +4 -3
- data/lib/parse/model/acl.rb +119 -21
- data/lib/parse/model/associations/belongs_to.rb +25 -3
- data/lib/parse/model/associations/collection_proxy.rb +146 -17
- data/lib/parse/model/associations/has_many.rb +49 -9
- data/lib/parse/model/associations/has_one.rb +3 -1
- data/lib/parse/model/associations/pointer_collection_proxy.rb +145 -17
- data/lib/parse/model/associations/relation_collection_proxy.rb +229 -38
- data/lib/parse/model/bytes.rb +13 -5
- data/lib/parse/model/classes/role.rb +95 -0
- data/lib/parse/model/classes/session.rb +304 -1
- data/lib/parse/model/classes/user.rb +104 -8
- data/lib/parse/model/core/actions.rb +287 -70
- data/lib/parse/model/core/builder.rb +100 -8
- data/lib/parse/model/core/create_lock.rb +27 -2
- data/lib/parse/model/core/describe.rb +2 -0
- data/lib/parse/model/core/fetching.rb +50 -5
- data/lib/parse/model/core/field_guards.rb +16 -8
- data/lib/parse/model/core/pluralized_aliases.rb +8 -4
- data/lib/parse/model/core/properties.rb +488 -39
- data/lib/parse/model/core/querying.rb +7 -0
- data/lib/parse/model/core/schema.rb +5 -3
- data/lib/parse/model/core/search_indexing.rb +63 -0
- data/lib/parse/model/core/vector_searchable.rb +35 -6
- data/lib/parse/model/file.rb +9 -2
- data/lib/parse/model/geopoint.rb +61 -13
- data/lib/parse/model/model.rb +160 -9
- data/lib/parse/model/object.rb +317 -18
- data/lib/parse/model/phone.rb +54 -5
- data/lib/parse/model/pointer.rb +40 -6
- data/lib/parse/model/push.rb +11 -1
- data/lib/parse/mongodb.rb +189 -61
- data/lib/parse/pipeline_security.rb +415 -26
- data/lib/parse/query/constraint.rb +31 -1
- data/lib/parse/query/constraints.rb +89 -59
- data/lib/parse/query/cursor.rb +3 -1
- data/lib/parse/query/operation.rb +62 -8
- data/lib/parse/query/ordering.rb +34 -6
- data/lib/parse/query.rb +1871 -167
- data/lib/parse/retrieval/agent_tool.rb +225 -8
- data/lib/parse/retrieval/benchmark.rb +149 -0
- data/lib/parse/retrieval/profiles.rb +320 -0
- data/lib/parse/retrieval/retriever.rb +10 -1
- data/lib/parse/retrieval.rb +2 -0
- data/lib/parse/schema/search_index_migrator.rb +23 -5
- data/lib/parse/schema.rb +74 -18
- data/lib/parse/stack/tasks.rb +6 -4
- data/lib/parse/stack/version.rb +1 -1
- data/lib/parse/stack.rb +91 -14
- data/lib/parse/two_factor_auth/user_extension.rb +14 -2
- data/lib/parse/two_factor_auth.rb +11 -0
- data/lib/parse/vector_search/hybrid.rb +110 -33
- data/lib/parse/vector_search/index_definition.rb +237 -0
- data/lib/parse/vector_search.rb +259 -23
- data/lib/parse/webhooks/payload.rb +136 -6
- data/lib/parse/webhooks/replay_protection.rb +58 -20
- data/lib/parse/webhooks.rb +715 -40
- metadata +8 -1
data/README.md
CHANGED
|
@@ -4,6 +4,16 @@
|
|
|
4
4
|
|
|
5
5
|
A full-featured Ruby client SDK for [Parse Server](http://parseplatform.org/). [parse-stack-next](https://github.com/neurosynq/parse-stack-next) is a Ruby client SDK, REST client, and Active Model ORM for [Parse Server](http://parseplatform.org/), combining a low-level API client, a query engine, an object-relational mapper (ORM), and a Cloud Code Webhooks rack application in a single gem.
|
|
6
6
|
|
|
7
|
+
## What's new in 5.8
|
|
8
|
+
|
|
9
|
+
- **5.8.0: MCP deployments can expose less than their users can read.** `Parse::Agent.new(fields: { Customer => %i[display_name timezone] })` narrows a class's `agent_fields` for one agent, so a user-facing assistant and an analytics endpoint in one process expose different subsets of the same model. It applies to queries, includes, aggregation, search, schema output, exports, and `semantic_search` text, and `query_class`/`count_objects`/`export_data` now refuse filtering or sorting on a hidden field. See [CHANGELOG.md](./CHANGELOG.md)
|
|
10
|
+
- **5.8.0: Supported deployment patterns.** `MCPRackApp.user_scoped` serves signed-in application users (no master-key fallback, server-pinned identity and tenant), and `MCPRackApp.master_analytics` serves a shared read-only analytics endpoint that requires an operator `principal_resolver`. Cancellations and approval replies are now bound to the session's owner, revocation intervals are documented, and orphaned subscriptions are reaped. See [CHANGELOG.md](./CHANGELOG.md)
|
|
11
|
+
- **5.8.0: Retrieval profiles for `semantic_search`.** Profiles you register on the server (for example `fast`, `balanced`, `precise`; none ship by default) compose hybrid search and reranking with budgets, spend accounting, and observable fallback; each call emits a sanitized `parse.retrieval.search` event, and `Parse::Retrieval::Benchmark` measures profiles on labeled cases. See [CHANGELOG.md](./CHANGELOG.md)
|
|
12
|
+
- **5.8.0: Vector index definitions from the model, with optional quantization.** `Parse::VectorSearch::IndexDefinition` generates the Atlas definition from your declarations (filters and tenant field included) with preview and diff, and `quantization: :scalar`/`:binary` on a `:vector` property shrinks index memory without changing stored data. Contextualized embedding batches now adapt to provider size limits. See [CHANGELOG.md](./CHANGELOG.md)
|
|
13
|
+
- **5.8.0: Two silent mismatches fixed; check whether you need to act.** Queries now send a property's declared `field:` name exactly as declared (`property :account_id, :string, field: :account_id` queries `account_id`, no longer `accountId`), so queries that matched nothing against underscore or mixed-case columns now return rows. Vector search, index auto-discovery, and drift checks now use the stored column of a multi-word `:vector` property: an Atlas vector index whose path is the Ruby name (`body_embedding`) must be recreated on the stored column (`bodyEmbedding`). Drift detection reports the mismatch. See [CHANGELOG.md](./CHANGELOG.md)
|
|
14
|
+
|
|
15
|
+
See [CHANGELOG.md](./CHANGELOG.md) for the full 5.8 entry.
|
|
16
|
+
|
|
7
17
|
## What's new in 5.7
|
|
8
18
|
|
|
9
19
|
- **5.7.6: `semantic_search` respects `agent_fields` for chunk content.** A class that embedded a field hidden by its `agent_fields` allowlist returned that field's text as chunk content. Chunk text now comes only from embedded fields the agent may read; a hidden text source is refused with `:field_denied` before the search runs. See [CHANGELOG.md](./CHANGELOG.md)
|
|
@@ -92,7 +102,7 @@ See [CHANGELOG.md](./CHANGELOG.md) for the full 5.1 entry, including breaking ch
|
|
|
92
102
|
- **MCP transport hardening** — Streamable HTTP `Mcp-Session-Id` header (renamed from `X-MCP-Session-Id`, **breaking**), `MCP-Protocol-Version` validation, `DELETE /` session termination, structured-content (`outputSchema`) on built-in tools, optional `health_path:` liveness probe
|
|
93
103
|
- **`Parse::GraphQL::TypeGenerator`** — generate `graphql-ruby` types directly from your `Parse::Object` subclasses (no Parse Server round-trip), with `:vector` columns surfaced as `[Float]` and association registries (`has_one_associations`, `has_many_associations`) populated at DSL time
|
|
94
104
|
- **LiveQuery promoted to stable** — the experimental warning is removed; `Parse.live_query_enabled = true` is retained as a network-egress safety toggle, not a stability gate
|
|
95
|
-
- **Server-version deprecation warning
|
|
105
|
+
- **Server-version deprecation warning**: a one-shot warning when connecting to a Parse Server older than the configured threshold (default `7.0.0`, override with `PARSE_DEPRECATED_SERVER_VERSION_BELOW`); silence with `Parse.suppress_server_version_warning = true`. The **supported baseline is Parse Server 9.x** (the SDK is developed and tested against a pinned `parse-server:9.10.3`); the default warning threshold is intentionally conservative so older deployments only get an advisory, not a hard break.
|
|
96
106
|
- **`mongo_relation_index :field, dedup: true`** — register a compound `{owningId, relatedId}` UNIQUE on relation join collections to prevent duplicate-pair subscriptions without breaking `has_many` semantics
|
|
97
107
|
|
|
98
108
|
See [CHANGELOG.md](./CHANGELOG.md) for the full 5.0 entry, including security-hardening notes and Ruby 3.x cleanup.
|
|
@@ -916,8 +926,8 @@ view = Parse.client.sdk_cache
|
|
|
916
926
|
store.verify_upstream_isolation!
|
|
917
927
|
```
|
|
918
928
|
|
|
919
|
-
**The two URLs must address different Redis databases.**
|
|
920
|
-
|
|
929
|
+
**The two URLs must address different Redis databases.** Through at least Parse
|
|
930
|
+
Server 9.10.3, a `_Role` write clears the cache with `FLUSHDB`, which on a shared
|
|
921
931
|
database deletes the SDK's cached responses and its create-locks along with it
|
|
922
932
|
([parse-server#10617](https://github.com/parse-community/parse-server/issues/10617)).
|
|
923
933
|
`verify_upstream_isolation!` detects this by scanning the SDK's own database
|
|
@@ -3863,8 +3873,8 @@ If you want to see if a particular field contains a specific Parse::Object (poin
|
|
|
3863
3873
|
q.where :field => Parse::Pointer.new("_User", "anObjectId")
|
|
3864
3874
|
# alias using subclass helper
|
|
3865
3875
|
q.where :field => Parse::User.pointer("anObjectId")
|
|
3866
|
-
# alias using
|
|
3867
|
-
q.where :user.
|
|
3876
|
+
# alias using the `pointer_id` constraint. We will infer :user maps to class "_User" (Parse::User)
|
|
3877
|
+
q.where :user.pointer_id => "anObjectId"
|
|
3868
3878
|
```
|
|
3869
3879
|
|
|
3870
3880
|
#### Less Than
|
|
@@ -3966,15 +3976,15 @@ Match arrays by their length:
|
|
|
3966
3976
|
|
|
3967
3977
|
```ruby
|
|
3968
3978
|
# Exact size
|
|
3969
|
-
q.where :tags.
|
|
3979
|
+
q.where :tags.array_size => 2 # arrays with exactly 2 elements
|
|
3970
3980
|
|
|
3971
3981
|
# Size comparisons
|
|
3972
|
-
q.where :tags.
|
|
3973
|
-
q.where :tags.
|
|
3974
|
-
q.where :tags.
|
|
3975
|
-
q.where :tags.
|
|
3976
|
-
q.where :tags.
|
|
3977
|
-
q.where :tags.
|
|
3982
|
+
q.where :tags.array_size => { gt: 3 } # size > 3
|
|
3983
|
+
q.where :tags.array_size => { gte: 2 } # size >= 2
|
|
3984
|
+
q.where :tags.array_size => { lt: 5 } # size < 5
|
|
3985
|
+
q.where :tags.array_size => { lte: 4 } # size <= 4
|
|
3986
|
+
q.where :tags.array_size => { ne: 0 } # size != 0
|
|
3987
|
+
q.where :tags.array_size => { gte: 2, lt: 10 } # range: 2 <= size < 10
|
|
3978
3988
|
|
|
3979
3989
|
# Empty/non-empty shortcuts (index-friendly)
|
|
3980
3990
|
q.where :tags.arr_empty => true # empty arrays (uses { field: [] })
|
|
@@ -4023,7 +4033,7 @@ All array constraints work with `has_many :through => :array` relations:
|
|
|
4023
4033
|
Product.query(:categories.set_equals => [cat1, cat2])
|
|
4024
4034
|
|
|
4025
4035
|
# Find products with more than 3 categories
|
|
4026
|
-
Product.query(:categories.
|
|
4036
|
+
Product.query(:categories.array_size => { gt: 3 })
|
|
4027
4037
|
```
|
|
4028
4038
|
|
|
4029
4039
|
**Note:** Array constraints using aggregation pipelines require MongoDB 3.6+.
|
|
@@ -4203,7 +4213,7 @@ Event.where(:updated_at.between_dates => [1.week.ago, Time.now])
|
|
|
4203
4213
|
```
|
|
4204
4214
|
|
|
4205
4215
|
#### Matches Object Id
|
|
4206
|
-
Sometimes you want to find rows where a particular Parse object exists. You can do so by passing a the Parse::Object subclass or a Parse::Pointer. In some cases you may only have the "objectId" of the record you are looking for. For convenience, you can also use the `
|
|
4216
|
+
Sometimes you want to find rows where a particular Parse object exists. You can do so by passing a the Parse::Object subclass or a Parse::Pointer. In some cases you may only have the "objectId" of the record you are looking for. For convenience, you can also use the `pointer_id` constraint. This will assume that the name of the field matches a particular Parse class you have defined. Assume the following:
|
|
4207
4217
|
|
|
4208
4218
|
```ruby
|
|
4209
4219
|
# where this Parse object equals the object in the column `field`.
|
|
@@ -4211,18 +4221,18 @@ q.where :field => Parse::Pointer("Field", "someObjectId")
|
|
|
4211
4221
|
# => "field":{"__type":"Pointer","className":"Field","objectId":"someObjectId"}}
|
|
4212
4222
|
|
|
4213
4223
|
# alias, shorthand when we infer `:field` maps to `Field` parse class.
|
|
4214
|
-
q.where :field.
|
|
4224
|
+
q.where :field.pointer_id => "someObjectId"
|
|
4215
4225
|
# => "field":{"__type":"Pointer","className":"Field","objectId":"someObjectId"}}
|
|
4216
4226
|
|
|
4217
4227
|
```
|
|
4218
4228
|
It is always important to be thoughtful in naming column names in associations as
|
|
4219
4229
|
close to their foreign Parse class names. This enables more expressive syntax while reducing
|
|
4220
|
-
code.
|
|
4230
|
+
code. `pointer_id` also accepts any object or pointer object. These are all equivalent:
|
|
4221
4231
|
|
|
4222
4232
|
```ruby
|
|
4223
4233
|
q.where :user => User.pointer("xyx123")
|
|
4224
|
-
q.where :user.
|
|
4225
|
-
q.where :user.
|
|
4234
|
+
q.where :user.pointer_id => "xyx123"
|
|
4235
|
+
q.where :user.pointer_id => User.pointer("xyx123")
|
|
4226
4236
|
# All produce
|
|
4227
4237
|
# => "user":{"__type":"Pointer","className":"_User","objectId":"xyx123"}}
|
|
4228
4238
|
```
|
|
@@ -4250,15 +4260,15 @@ In some cases, you do not have the Parse object, but you have its `objectId`. Yo
|
|
|
4250
4260
|
|
|
4251
4261
|
```ruby
|
|
4252
4262
|
# shorthand if you are using convention. Will infer class `Artist`
|
|
4253
|
-
Song.all :artist.
|
|
4263
|
+
Song.all :artist.pointer_id => artist_id
|
|
4254
4264
|
|
|
4255
4265
|
# other approaches, same result
|
|
4256
4266
|
Song.all :artist => Artist.pointer(artist_id)
|
|
4257
4267
|
Song.all :artist => Parse::Pointer.new("Artist", artist_id)
|
|
4258
4268
|
|
|
4259
|
-
#
|
|
4269
|
+
# `pointer_id` safely accepts pointers and strings for these kinds of API patterns
|
|
4260
4270
|
def find_songs(artist)
|
|
4261
|
-
Song.all :artist.
|
|
4271
|
+
Song.all :artist.pointer_id => artist
|
|
4262
4272
|
end
|
|
4263
4273
|
|
|
4264
4274
|
# all ok
|
|
@@ -5803,7 +5813,7 @@ Event.query(:event_date.gt => Time.now).results_direct
|
|
|
5803
5813
|
Event.query(:event_date.gte => start_date, :event_date.lte => end_date).results_direct
|
|
5804
5814
|
|
|
5805
5815
|
# Array operators
|
|
5806
|
-
Song.query(:tags.
|
|
5816
|
+
Song.query(:tags.array_size => 3).results_direct
|
|
5807
5817
|
Song.query(:tags.contains_all => ["rock", "classic"]).results_direct
|
|
5808
5818
|
Song.query(:tags.empty_or_nil => true).results_direct
|
|
5809
5819
|
|
|
@@ -6091,6 +6101,12 @@ Parse.client.authorization.invalidate(token)
|
|
|
6091
6101
|
Parse.client.authorization.invalidate_user_roles(user_id)
|
|
6092
6102
|
```
|
|
6093
6103
|
|
|
6104
|
+
A custom identity plane that implements neither `invalidate_value` nor
|
|
6105
|
+
generation support (`generation`, `generation_current?`, `bump_generation`)
|
|
6106
|
+
can only forget one token at a time. Revoking every session
|
|
6107
|
+
of a user (password change, account deletion, `logout_all!`, a batch destroy of
|
|
6108
|
+
sessions) is then bounded by `identity_cache_ttl` rather than immediate.
|
|
6109
|
+
|
|
6094
6110
|
Both caches default to per-process memory. With a keyspaced
|
|
6095
6111
|
`Parse::Cache::Redis` you can move them to the shared backend so every worker
|
|
6096
6112
|
resolves against the same view, and let webhook triggers invalidate them
|
|
@@ -6473,7 +6489,7 @@ export PARSE_TEST_SERVER_URL=http://localhost:29337/parse
|
|
|
6473
6489
|
export PARSE_TEST_APP_ID=psnextItAppId
|
|
6474
6490
|
export PARSE_TEST_API_KEY=psnext-it-rest-key
|
|
6475
6491
|
export PARSE_TEST_MASTER_KEY=psnextItMasterKey
|
|
6476
|
-
export PARSE_TEST_MONGO_URI="mongodb://admin:password@localhost:29017/parse_stack_next_it?authSource=admin"
|
|
6492
|
+
export PARSE_TEST_MONGO_URI="mongodb://admin:password@localhost:29017/parse_stack_next_it?authSource=admin&directConnection=true"
|
|
6477
6493
|
export PARSE_TEST_REDIS_URL=redis://localhost:29379/0
|
|
6478
6494
|
export PARSE_TEST_LIVE_QUERY_URL=ws://localhost:29337
|
|
6479
6495
|
export ATLAS_URI="mongodb://localhost:29020/parse_atlas_test?directConnection=true"
|
data/docs/TEST_SERVER.md
CHANGED
|
@@ -62,8 +62,8 @@ default.
|
|
|
62
62
|
- **`PSNEXT_PREFIX`** (default `psnext-it`) names the Compose project and every
|
|
63
63
|
container. Set it (e.g. `PSNEXT_PREFIX=psnext-ci`) to run a second, fully
|
|
64
64
|
separate copy.
|
|
65
|
-
- **Versions**: Parse Server is pinned to `parseplatform/parse-server:9.
|
|
66
|
-
(see `scripts/docker/Dockerfile.parse`), MongoDB `mongo:
|
|
65
|
+
- **Versions**: Parse Server is pinned to `parseplatform/parse-server:9.10.3`
|
|
66
|
+
(see `scripts/docker/Dockerfile.parse`), MongoDB `mongo:9` (`MONGO_VERSION`), Redis
|
|
67
67
|
`redis:7-alpine`, Dashboard `parseplatform/parse-dashboard:9`.
|
|
68
68
|
- **Database**: Parse uses `parse_stack_next_it`.
|
|
69
69
|
|
data/docs/acl_clp_guide.md
CHANGED
|
@@ -491,6 +491,13 @@ described in §7. From a security standpoint, an Atlas Search call
|
|
|
491
491
|
with a session token is treated like a `Parse::Query` with a session
|
|
492
492
|
token — same scoping, same field stripping.
|
|
493
493
|
|
|
494
|
+
Stripping a protected field from results does not stop it from
|
|
495
|
+
deciding which documents match. A scoped `Parse::AtlasSearch.search`
|
|
496
|
+
is therefore refused (`Parse::CLPScope::Denied`) when `fields:` names a
|
|
497
|
+
field protected for the caller, or when it names no fields (a search
|
|
498
|
+
over every column) while the caller has any protected fields. Pass
|
|
499
|
+
`fields:` with the fields to search.
|
|
500
|
+
|
|
494
501
|
The `$search` stage itself runs on the Atlas Search index and is not
|
|
495
502
|
filtered by ACL. The ACL filter is applied as a `$match` stage by
|
|
496
503
|
`Parse::ACLScope` after `$search`, before results are returned. If
|
|
@@ -209,7 +209,7 @@ Parse::AtlasSearch::IndexCatalog.create_index(
|
|
|
209
209
|
fields: [
|
|
210
210
|
{
|
|
211
211
|
type: "vector",
|
|
212
|
-
path: "
|
|
212
|
+
path: "bodyEmbedding", # the STORED column, not the Ruby name
|
|
213
213
|
numDimensions: 1536,
|
|
214
214
|
similarity: "cosine",
|
|
215
215
|
},
|
|
@@ -221,6 +221,12 @@ Parse::AtlasSearch::IndexCatalog.create_index(
|
|
|
221
221
|
)
|
|
222
222
|
```
|
|
223
223
|
|
|
224
|
+
The vector `path` is the column the property is stored under:
|
|
225
|
+
`property :body_embedding, :vector` is saved as `bodyEmbedding` (or under an
|
|
226
|
+
explicit `field:` alias). `find_similar`, hybrid search, index discovery, and
|
|
227
|
+
drift checks all use that stored name; `Parse::VectorSearch::IndexDefinition`
|
|
228
|
+
generates it for you.
|
|
229
|
+
|
|
224
230
|
Including `_rperm` as a filter field lets the per-row ACL match
|
|
225
231
|
short-circuit at the index level — strongly recommended for any
|
|
226
232
|
field that ACL-scoped agents will search against.
|
|
@@ -241,6 +247,91 @@ indexes for one whose definition covers the requested `path`. The
|
|
|
241
247
|
first match wins; pass `index:` explicitly when you have more than
|
|
242
248
|
one covering index and want a specific one.
|
|
243
249
|
|
|
250
|
+
### Generating the definition from the model
|
|
251
|
+
|
|
252
|
+
Writing the definition by hand lets it drift from the model: a changed
|
|
253
|
+
`dimensions:`, a new `agent_searchable` filter field, or a tenant scope
|
|
254
|
+
the index does not cover. `Parse::VectorSearch::IndexDefinition` derives
|
|
255
|
+
the definition from the model's own declarations instead:
|
|
256
|
+
|
|
257
|
+
* the `:vector` property's path, `dimensions:`, `similarity:` (`cosine`
|
|
258
|
+
when undeclared), and optional `quantization:`;
|
|
259
|
+
* every `agent_searchable filter_fields:` entry as a `type: "filter"` path
|
|
260
|
+
(pointer fields use their `_p_<column>` storage path);
|
|
261
|
+
* the `agent_tenant_scope` field, which retrieval folds into
|
|
262
|
+
`$vectorSearch.filter` on every scoped query.
|
|
263
|
+
|
|
264
|
+
Output is deterministic (vector entry first, filters sorted by path), so
|
|
265
|
+
it diffs cleanly in review.
|
|
266
|
+
|
|
267
|
+
```ruby
|
|
268
|
+
class Document < Parse::Object
|
|
269
|
+
property :category, :string
|
|
270
|
+
property :embedding, :vector, dimensions: 1024, similarity: :dotProduct
|
|
271
|
+
agent_searchable field: :embedding, filter_fields: %i[category]
|
|
272
|
+
|
|
273
|
+
# Declare the index; its definition is generated, not written by hand.
|
|
274
|
+
vector_search_index "document_vec"
|
|
275
|
+
end
|
|
276
|
+
|
|
277
|
+
# Preview without touching Atlas.
|
|
278
|
+
Parse::VectorSearch::IndexDefinition.build(Document)
|
|
279
|
+
Parse::Schema.vector_index_definition(Document) # same thing
|
|
280
|
+
Parse::VectorSearch::IndexDefinition.preview(Document, name: "document_vec")
|
|
281
|
+
|
|
282
|
+
# Compare against what is deployed (a definition or a $listSearchIndexes entry).
|
|
283
|
+
live = Parse::AtlasSearch::IndexCatalog.find_vector_index("Document", field: :embedding)
|
|
284
|
+
Parse::VectorSearch::IndexDefinition.diff(Parse::Schema.vector_index_definition(Document), live)
|
|
285
|
+
# => { in_sync: false,
|
|
286
|
+
# vector: { "numDimensions" => { declared: 1024, live: 1536 } },
|
|
287
|
+
# filters_missing: ["category"], filters_extra: [] }
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
Applying stays explicit. `vector_search_index` declarations join the
|
|
291
|
+
model's `mongo_search_index` declarations in
|
|
292
|
+
`Parse::Schema::SearchIndexMigrator`, which plans first and only mutates
|
|
293
|
+
Atlas when asked; a drifted index is reported, not rebuilt, unless you
|
|
294
|
+
pass `update: true`. The definition is generated when the migrator plans,
|
|
295
|
+
so `agent_searchable` and `agent_tenant_scope` can be declared in any
|
|
296
|
+
order.
|
|
297
|
+
|
|
298
|
+
```ruby
|
|
299
|
+
Document.search_indexes_plan # :to_create / :in_sync / :drifted / :orphans
|
|
300
|
+
Document.apply_search_indexes! # creates missing indexes only
|
|
301
|
+
Document.apply_search_indexes!(update: true) # also rebuilds drifted ones
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
### Index-side quantization
|
|
305
|
+
|
|
306
|
+
Atlas can quantize a float vector field when it builds the index, which
|
|
307
|
+
shrinks the index Atlas keeps in memory. Declare it per property; it is
|
|
308
|
+
off by default:
|
|
309
|
+
|
|
310
|
+
```ruby
|
|
311
|
+
property :embedding, :vector, dimensions: 1024, quantization: :scalar # or :binary
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
The generated definition then carries `"quantization": "scalar"` (or
|
|
315
|
+
`"binary"`) on the vector field. Only the index changes: stored vectors and
|
|
316
|
+
the SDK write path stay full precision, so turning it on or off is an index
|
|
317
|
+
rebuild, not a re-embed.
|
|
318
|
+
|
|
319
|
+
| Setting | Index memory | Recall |
|
|
320
|
+
|---------|--------------|--------|
|
|
321
|
+
| none (default) | full float vectors | baseline |
|
|
322
|
+
| `:scalar` | roughly 4x smaller | small loss for most embedding models |
|
|
323
|
+
| `:binary` | roughly 32x smaller | larger loss; Atlas rescoring recovers part of it |
|
|
324
|
+
|
|
325
|
+
Measure recall on your own queries before adopting `:binary`; the right
|
|
326
|
+
choice depends on the embedding model and the collection. For small
|
|
327
|
+
collections the memory saving rarely matters.
|
|
328
|
+
|
|
329
|
+
First-query drift verification also checks quantization: an index whose
|
|
330
|
+
`quantization` differs from the property's declaration (an absent value
|
|
331
|
+
counts as none on either side) is reported under
|
|
332
|
+
`Parse::VectorSearch.index_drift_policy` like a dimension or similarity
|
|
333
|
+
mismatch.
|
|
334
|
+
|
|
244
335
|
---
|
|
245
336
|
|
|
246
337
|
## Running similarity queries: `find_similar`
|
|
@@ -302,6 +393,9 @@ index's `latestDefinition` against the model declaration:
|
|
|
302
393
|
(usually an index that predates a model change).
|
|
303
394
|
* `similarity` vs the property's declared `similarity:` (checked only
|
|
304
395
|
when both sides declare one).
|
|
396
|
+
* `quantization` vs the property's declared `quantization:` (absent on
|
|
397
|
+
either side means none, so an index quantized without a declaration
|
|
398
|
+
is drift too).
|
|
305
399
|
* When the class registers an `agent_tenant_scope`, the scope field
|
|
306
400
|
must appear among the index's `type: "filter"` paths — without it,
|
|
307
401
|
every tenant-scoped `$vectorSearch.filter` fails Atlas-side at
|
|
@@ -505,18 +599,27 @@ Two things to know:
|
|
|
505
599
|
one-chunk document. Registering a context model as an `embed` provider
|
|
506
600
|
works, but the stored vectors carry no surrounding-document context.
|
|
507
601
|
Use `embed_chunks` when chunk-level context is the point.
|
|
508
|
-
* **Request sizing is
|
|
509
|
-
at 1,000 documents, 16,000 chunks, and 120k tokens.
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
602
|
+
* **Request sizing adapts; it is not exact token counting.** Voyage caps
|
|
603
|
+
one request at 1,000 documents, 16,000 chunks, and 120k input tokens.
|
|
604
|
+
`embed_chunks` (and `embed_text` on a context model) packs whole
|
|
605
|
+
documents into requests that stay within the document cap, a chunk
|
|
606
|
+
count whose **response** fits the SDK's response-size cap, and an
|
|
607
|
+
**estimated** 120k-token budget. The SDK has no tokenizer, so the
|
|
608
|
+
estimate assumes three bytes per token, which over-counts typical
|
|
609
|
+
English text and packs conservatively. A document's chunks always
|
|
610
|
+
travel together.
|
|
611
|
+
When Voyage still rejects a request as too large (its "batch size" or
|
|
612
|
+
"max allowed tokens per submitted batch" errors, or an HTTP 413), the
|
|
613
|
+
SDK halves that request by document and resends each half, keeping
|
|
614
|
+
the returned vectors aligned with your input. Only those positively
|
|
615
|
+
identified size errors trigger a split; any other 400 is raised as is,
|
|
616
|
+
with the provider's message on `BadRequestError#detail`. A document
|
|
617
|
+
that is rejected as too large even on its own raises a
|
|
618
|
+
`BadRequestError` naming its index, since no split can fix it: break
|
|
619
|
+
it into fewer or shorter chunks. A single chunk longer than the
|
|
620
|
+
model's context window ("tokens in an example exceeds the context
|
|
621
|
+
length") is also raised directly. Context models default to
|
|
622
|
+
`embed_batch_size: 32` for `embed_text` batches.
|
|
520
623
|
|
|
521
624
|
---
|
|
522
625
|
|
|
@@ -678,6 +781,71 @@ adapters implement only the network call (`#rerank_scores`).
|
|
|
678
781
|
> rate-limited tool error). Admin agents are exempt; direct
|
|
679
782
|
> `find_similar` / `retrieve` callers are not metered.
|
|
680
783
|
|
|
784
|
+
### Retrieval profiles for `semantic_search` (5.8)
|
|
785
|
+
|
|
786
|
+
An agent can choose among a few server-configured retrieval strategies by
|
|
787
|
+
name, without ever choosing a provider, endpoint, or credential. Register
|
|
788
|
+
rerankers by name, then profiles that reference them:
|
|
789
|
+
|
|
790
|
+
```ruby
|
|
791
|
+
Parse::Retrieval.register_reranker(:voyage,
|
|
792
|
+
Parse::Retrieval::Reranker::Voyage.new(api_key: ENV.fetch("VOYAGE_API_KEY"), model: "rerank-3-lite"))
|
|
793
|
+
|
|
794
|
+
Parse::Retrieval::Profiles.register(:fast, k: 5, max_k: 10)
|
|
795
|
+
Parse::Retrieval::Profiles.register(:balanced, k: 8, hybrid: true)
|
|
796
|
+
Parse::Retrieval::Profiles.register(:precise,
|
|
797
|
+
k: 8, reranker: :voyage,
|
|
798
|
+
rerank_candidates: 30, # documents retrieved and sent to the reranker (max 100)
|
|
799
|
+
rerank_top_n: 8, # documents kept after reranking
|
|
800
|
+
rerank_max_document_chars: 4_000, # each document's text is cut before it leaves the process
|
|
801
|
+
rerank_timeout: 5, # seconds
|
|
802
|
+
on_rerank_failure: :fallback, # or :raise
|
|
803
|
+
max_total_tokens: 8_000) # default response budget
|
|
804
|
+
|
|
805
|
+
agent.execute(:semantic_search, class_name: "Article", query: "refund policy", profile: "precise")
|
|
806
|
+
```
|
|
807
|
+
|
|
808
|
+
* **Validated at registration.** An unknown option, an unregistered reranker,
|
|
809
|
+
`k` above `max_k`, or a non-positive budget raises `ArgumentError` at boot.
|
|
810
|
+
An unknown profile name at call time is refused with the list of available
|
|
811
|
+
profiles.
|
|
812
|
+
* **Field-safe reranking.** The reranker receives the same text source as
|
|
813
|
+
chunk content, which `semantic_search` restricts to fields the agent may
|
|
814
|
+
read (its effective `agent_fields`, including any per-agent `fields:`
|
|
815
|
+
narrowing), cut to `rerank_max_document_chars`.
|
|
816
|
+
* **Spend.** Estimated rerank tokens are charged to the same per-tenant
|
|
817
|
+
`SpendCap` budget as the query embedding (admin agents are exempt).
|
|
818
|
+
* **Fallback is observable.** On a reranker timeout or provider error,
|
|
819
|
+
`:fallback` keeps the retrieval order and adds `rerank_fallback: true` and
|
|
820
|
+
`rerank_fallback_reason` to the result; `:raise` fails the call.
|
|
821
|
+
* **Defaults are unchanged.** Without `profile:` the tool behaves as before.
|
|
822
|
+
|
|
823
|
+
Each call emits one `parse.retrieval.search` notification with the profile,
|
|
824
|
+
`k`, candidate count, rerank stats (`used`, `documents`, `chars`,
|
|
825
|
+
`tokens_estimated`, `duration_ms`, `fallback`, `fallback_reason`), chunk and
|
|
826
|
+
document counts, budget drops, and timings. It never carries the query,
|
|
827
|
+
document text, field values, URLs, or credentials, and `tokens_estimated` is
|
|
828
|
+
the SDK's estimate, not provider-reported usage.
|
|
829
|
+
|
|
830
|
+
**Measuring profiles.** `Parse::Retrieval::Benchmark` scores profiles on a
|
|
831
|
+
labeled case set (recall@k, MRR, hit rate, mean and p95 latency, estimated
|
|
832
|
+
rerank tokens), overall and per tag. Cases may list `forbidden` ids that must
|
|
833
|
+
never be returned (restrictive ACLs, other tenants); any such hit is reported
|
|
834
|
+
as a violation.
|
|
835
|
+
|
|
836
|
+
```ruby
|
|
837
|
+
cases = Parse::Retrieval::Benchmark.load_cases("eval/cases.json")
|
|
838
|
+
runner = Parse::Retrieval::Benchmark.semantic_search_runner(agent, class_name: "Article")
|
|
839
|
+
# nil is the baseline (no profile); its results are reported under "default".
|
|
840
|
+
report = Parse::Retrieval::Benchmark.run(cases: cases, profiles: [nil, "fast", "precise"], runner: runner)
|
|
841
|
+
report["default"] # => the baseline, for comparison
|
|
842
|
+
report["precise"] # => { recall_at_k:, mrr:, hit_rate:, violations:, mean_ms:, p95_ms:, ... }
|
|
843
|
+
report["precise"][:by_tag]["long_document"]
|
|
844
|
+
```
|
|
845
|
+
|
|
846
|
+
Run it against your own data and Atlas index before recommending a profile;
|
|
847
|
+
the SDK does not ship measured defaults.
|
|
848
|
+
|
|
681
849
|
### Chunkers
|
|
682
850
|
|
|
683
851
|
The default is a fixed-size sliding window with overlap. Subclass
|
data/docs/caching.md
CHANGED
|
@@ -358,6 +358,56 @@ Two behaviors to know before you rely on these:
|
|
|
358
358
|
triggers depend on. The deprecated `Parse::AtlasSearch::Session.invalidate` /
|
|
359
359
|
`.invalidate_user_roles` forms still work through 5.x: they delegate to the
|
|
360
360
|
default client's context, so they can only ever address `Parse.client`.
|
|
361
|
+
* A custom identity plane you write yourself needs `get`, `set`, and
|
|
362
|
+
`invalidate`. Revoking one token (logout, `Parse::Session#destroy` on a
|
|
363
|
+
session that carries its token) works with those alone. Revoking every token
|
|
364
|
+
of a user (password change, account deletion, `logout_all!`, destroying a
|
|
365
|
+
session fetched without its token, or a batch `Array#destroy` of sessions or
|
|
366
|
+
users) also needs either `invalidate_value(user_id)` or generation support
|
|
367
|
+
(`generation`, `generation_current?`, and `bump_generation`), because the
|
|
368
|
+
plane is keyed by token. A plane with neither keeps resolving
|
|
369
|
+
those tokens until their cached entries expire, so revocation there is
|
|
370
|
+
bounded by `identity_cache_ttl`, not immediate. Lower the TTL if that window
|
|
371
|
+
is too long.
|
|
372
|
+
* A session or user delete drops its cached identity when the delete succeeds
|
|
373
|
+
or reports the row already gone ("object not found"); a denied delete leaves
|
|
374
|
+
the cache alone. A session fetched without its token is looked up first
|
|
375
|
+
(with the master key when the client has one, otherwise with the delete's
|
|
376
|
+
`session:`). When the lookup finds no row (the session is already gone or
|
|
377
|
+
not visible), the delete uses the owner recorded when a `_Session` query or
|
|
378
|
+
`Parse::Session.session` last fetched that session through the same client.
|
|
379
|
+
Only the owner's user id is recorded, never the token, for as long as an
|
|
380
|
+
identity entry lives. The records sit in their own store
|
|
381
|
+
(`client.authorization.session_owner_cache`, process-local by default, set
|
|
382
|
+
a shared store to share them), never in the identity plane, so no session
|
|
383
|
+
token can read one. With no
|
|
384
|
+
recorded owner, a batch delete that reports the row deleted or "object not
|
|
385
|
+
found" clears the client's identity and role cache, and so does a lookup
|
|
386
|
+
that fails outright. Either reset happens at most once every 5 seconds per
|
|
387
|
+
client, so an endpoint that deletes caller-supplied session ids cannot flush
|
|
388
|
+
a shared plane on every request. A single `destroy` cannot tell "already
|
|
389
|
+
gone" from "denied", so it uses the recorded owner but never resets. The
|
|
390
|
+
residual window is a session deleted elsewhere whose owner this process
|
|
391
|
+
never recorded: its cached token resolves until the next reset or until
|
|
392
|
+
`identity_cache_ttl`.
|
|
393
|
+
* A token resolution that is in flight when an invalidation lands does not
|
|
394
|
+
cache its answer, so it cannot put a just-revoked token back. Every
|
|
395
|
+
invalidation moves a marker before it drops entries, and the resolver
|
|
396
|
+
checks the marker before writing and again after, evicting its own write
|
|
397
|
+
when it moved. On the Redis identity plane the marker is a random nonce
|
|
398
|
+
shared by every process, replaced on each invalidation, so this also holds
|
|
399
|
+
for invalidations made by other processes, including a full reset. A custom
|
|
400
|
+
plane gets the same cross-process guarantee by implementing
|
|
401
|
+
`invalidation_nonce` (return the current marker String, or nil when none is
|
|
402
|
+
set) and `bump_invalidation_nonce` (store and return a new random String).
|
|
403
|
+
A custom plane with generation support but no nonce shares a generation
|
|
404
|
+
counter instead. That covers single-token and per-user invalidations made
|
|
405
|
+
by other processes, but not a full reset if the plane's `clear` also deletes
|
|
406
|
+
its generation counters: the counter restarts at a value an in-flight
|
|
407
|
+
lookup may already hold, so a lookup racing another process's reset can
|
|
408
|
+
keep its answer until `identity_cache_ttl`. A plane with neither gets the
|
|
409
|
+
guarantee within one process only, and across processes it is bounded by
|
|
410
|
+
`identity_cache_ttl`.
|
|
361
411
|
|
|
362
412
|
On a Redis outage these planes behave differently from the response cache.
|
|
363
413
|
The response cache degrades to a passthrough request; the identity and role
|
data/docs/client_sdk_guide.md
CHANGED
|
@@ -1176,6 +1176,17 @@ server-side on every event before it goes out the WebSocket — Bob will
|
|
|
1176
1176
|
not receive an event for an ACL-private row Alice creates, even if his
|
|
1177
1177
|
subscription matches the `where` clause.
|
|
1178
1178
|
|
|
1179
|
+
> **Role-based CLP grants need a server option.** Parse Server does not
|
|
1180
|
+
> resolve the subscriber's roles when LiveQuery evaluates Class Level
|
|
1181
|
+
> Permissions unless `enableLiveQueryClassLevelPermissionRoles: true` is
|
|
1182
|
+
> set (env `PARSE_SERVER_ENABLE_LIVE_QUERY_CLASS_LEVEL_PERMISSION_ROLES`,
|
|
1183
|
+
> available from Parse Server 9.10.3, default `false`). With the default,
|
|
1184
|
+
> a CLP such as `"find" => { "role:Editor" => true }` admits role members
|
|
1185
|
+
> over REST but rejects their LiveQuery subscription. Deployments that
|
|
1186
|
+
> grant LiveQuery access through `role:` entries in CLP must turn it on.
|
|
1187
|
+
> Object ACL `role:` entries are honored either way. The SDK test stack
|
|
1188
|
+
> enables it in `scripts/start-parse.sh`.
|
|
1189
|
+
|
|
1179
1190
|
> **Master-key authorization is per-CONNECTION, not per-subscription.**
|
|
1180
1191
|
> Parse Server resolves master-key (ACL/CLP-bypass) authorization once,
|
|
1181
1192
|
> from the connect frame; once set, EVERY subscription on that socket
|