parse-stack-next 5.0.1 → 5.1.0

data/docs/atlas_vector_search_guide.md

@@ -1,11 +1,19 @@
 # Atlas Vector Search Guide
 
-Parse Stack v5.0 ships first-class support for MongoDB Atlas
-`$vectorSearch` against Parse classes. This guide covers the full
-surface: declaring `:vector` properties, registering embedding
-providers, running `find_similar` queries, the `embed` write-side
-macro, Atlas index management, AS::N telemetry, and the constraint and
-logging behavior callers need to know about.
+Parse Stack ships first-class support for MongoDB Atlas `$vectorSearch`
+against Parse classes. This guide covers the full surface: declaring
+`:vector` properties, registering embedding providers, running
+`find_similar` queries, the `embed` and `embed_image` write-side
+macros, Atlas index management, AS::N telemetry, and the constraint
+and logging behavior callers need to know about.
+
+v5.0 introduced the text-embedding path; v5.1 adds image embedding via
+the new `embed_image` macro, `Voyage#embed_image`
+(`voyage-multimodal-3`, 1024-dim), and `Cohere#embed_image`
+(`embed-v4.0`, 1536-dim). Image inputs are URL-only in v5.1 (the SDK
+forwards the file URL to the provider; the SDK does not fetch image
+bytes) and are gated behind an explicit operator opt-in plus a CDN
+allowlist — see §Image embedding below.
 
 For the underlying mongo-direct enforcement model that vector search
 inherits, see [mongodb_direct_guide.md](./mongodb_direct_guide.md).
@@ -95,7 +103,7 @@ traces. The wire payload itself is unchanged.
 
 ## Registering an embedding provider
 
-`Parse::Embeddings` is a pluggable registry. v5.0 ships seven built-in
+`Parse::Embeddings` is a pluggable registry. v5.1 ships seven built-in
 providers:
 
 * `Parse::Embeddings::OpenAI` — text-only. `text-embedding-3-small`
@@ -106,16 +114,18 @@ providers:
   `embed-multilingual-v3.0`, and `-light-v3.0` siblings; 1024 / 384 dim)
   plus `embed-v4.0` (1536 native, 128k token context, Matryoshka-
   truncatable to {256, 512, 1024, 1536} via `dimensions:`). `embed-v4.0`
-  is Cohere's text+image multimodal endpoint at the network boundary;
-  this release wires the **text path only** — `embed_image` lands in
-  v5.1.
+  is Cohere's text+image multimodal endpoint; the text path routes
+  through `/v1/embed` and the v5.1 image path routes through
+  `/v2/embed` with OpenAI-style nested
+  `{ type: "image_url", image_url: { url: ... } }` content rows.
 * `Parse::Embeddings::Voyage` — voyage-4 family (`voyage-4-large` 2048,
   Matryoshka; `voyage-4` 1024; `voyage-4-lite` 512; `voyage-4-nano` 256),
   voyage-3 family, domain models (`voyage-code-3`, `voyage-finance-2`,
   `voyage-law-2`), and `voyage-multimodal-3` (1024-dim, 32k token
   context, routes to `/v1/multimodalembeddings` with the wrapped
-  `{inputs: [{content: [{type: "text", text: ...}]}]}` envelope). Text
-  inputs only in v5.0 — image content rows land in v5.1.
+  `{inputs: [{content: [{type: "text", text: ...}]}]}` envelope for
+  text and `{type: "image_url", image_url: <url>}` content rows for
+  the v5.1 `embed_image` path).
 * `Parse::Embeddings::Jina` — `jina-embeddings-v3` (1024, Matryoshka
   32–1024), `jina-embeddings-v4` (2048, Matryoshka), v5 family
   (`jina-embeddings-v5-text-{small,nano}`,
@@ -339,7 +349,7 @@ Mechanics:
   skipped. If every source is blank, the target and digest are both
   cleared on save.
 
-### Single vector per record (v5.0)
+### Single vector per record
 
 `embed` produces exactly one vector per record. There is no built-in
 chunker. Long source text whose concatenation exceeds the provider's
@@ -347,7 +357,7 @@ per-call token budget will be truncated provider-side, and the
 resulting vector will represent only the leading portion of the
 document.
 
-For long-form content in v5.0, two options:
+For long-form content, two options:
 
 1. **Pre-chunk client-side** and write each chunk as its own
    `Parse::Object` record with its own `embed` declaration.
@@ -357,16 +367,102 @@ For long-form content in v5.0, two options:
    parents as needed.
 
 A built-in chunker plus a `semantic_search` agent tool are scheduled
-for v5.1.
+for a future release.
 
-### Re-embedding existing rows
+---
+
+## Image embedding: `embed_image` macro (v5.1)
+
+`embed_image` is the image-source counterpart to `embed`. The source
+property must be `:file`-typed; the target must be a `:vector` property
+whose declared `provider:` supports multimodal input (currently
+`:voyage` with `voyage-multimodal-3`, or `:cohere` with `embed-v4.0`).
+
+```ruby
+class Post < Parse::Object
+  property :cover_image,            :file
+  property :cover_image_embedding,  :vector,
+           dimensions: 1024,
+           provider:   :voyage,
+           model:      "voyage-multimodal-3"
+
+  embed_image :cover_image, into: :cover_image_embedding
+end
+```
+
+### Operator setup (required before any save)
+
+Image embedding hands an attacker-influenced URL (a user-uploaded
+`Parse::File`, a chat message, an agent tool argument) to a third-party
+provider that will issue an HTTP request from its own network. The
+provider's fetch happens after SDK-side validation, so DNS rebinding
+and redirect-following are residual risks the SDK cannot eliminate.
+
+The setup must happen in this exact order — skipping (1) or (2) raises
+a typed error at save time with a message naming the missing
+prerequisite:
+
+```ruby
+# (1) Declare which CDNs the validator will accept. Empty allowlist
+# denies every host — opposite of Parse::File.allowed_remote_hosts.
+Parse::Embeddings.allowed_image_hosts = [
+  ".cloudfront.net",                # suffix match (leading ".")
+  "files.example.com",              # exact match
+]
+
+# (2) Sentinel-gated opt-in. Only the exact frozen String unlocks;
+# `true`, `"true"`, `1`, or any other value raises
+# Parse::Embeddings::ConfirmationRequired.
+Parse::Embeddings.trust_provider_url_fetch = "PROVIDER_EGRESS_VERIFIED"
+
+# (3) Declare embed_image on the model.
+class Post < Parse::Object
+  embed_image :cover_image, into: :cover_image_embedding
+end
+```
+
+### URL validator (`Parse::Embeddings.validate_image_url!`)
+
+Every `embed_image` save path routes through
+`Parse::Embeddings.validate_image_url!(url, allow_insecure:)`, which
+runs layered cheap-first checks: sentinel set, `https://` (or
+`http://` with `allow_insecure: true`), no userinfo, host not an
+obfuscated-IP form (`0x7f.0.0.1`, `127.1`, `2130706433`), host in the
+allowlist, port in `Parse::File.allowed_remote_ports`, host resolves
+only to public addresses (delegated to
+`Parse::File.assert_host_allowed!` so the SSRF mechanism is shared
+with `Parse::File`, not parallelized). Failures raise
+`Parse::Embeddings::InvalidImageURL` with a `:reason` Symbol
+(`:scheme`, `:port`, `:userinfo`, `:host_blocked`,
+`:host_not_allowlisted`, `:parse`).
+
+### Save-side semantics
+
+* Digest is the **SHA-256 of the URL String**, not the file bytes.
+  Replacing the `Parse::File` with one pointing at a different URL
+  re-embeds; resaving the same URL is a no-op (zero provider calls).
+  Parse-managed file URLs are stable unless overwritten in place — if
+  you PUT-replace bytes at the same URL (S3 without renaming), null
+  the digest field to force re-embed.
+* The same `EmbedManaged` write-guard applies: direct assignment to
+  the managed vector raises `ProtectedFieldError`. The write path is
+  the only way to populate the target vector.
+* `embed` and `embed_image` can co-declare on the same record
+  (different source properties → different `:vector` targets), so a
+  record can have one text-embedding column and one image-embedding
+  column queried by separate Atlas vectorSearch indexes.
+
+---
+
+## Re-embedding existing rows
 
 Changing `model:`, `dimensions:`, or `provider:` on an existing
-`:vector` property is a migration. Workflow:
+`:vector` property is a migration regardless of whether the source is
+text or images. Workflow:
 
 1. Add the new property alongside the old one
-   (`property :body_embedding_v2, :vector, ...`) and an `embed` block
-   targeting it.
+   (`property :body_embedding_v2, :vector, ...`) and an `embed` or
+   `embed_image` block targeting it.
 2. Backfill: iterate existing rows, force a save (or null+save) to
    trigger the new directive. The old field stays valid for reads.
 3. Once backfill completes, deploy a new vectorSearch index covering
@@ -374,7 +470,10 @@ Changing `model:`, `dimensions:`, or `provider:` on an existing
 4. Drop the old property.
 
 Do NOT mutate the model in place — the digest mechanism will see
-unchanged source text and skip recompute, leaving stale vectors.
+unchanged source text / unchanged source URL and skip recompute,
+leaving stale vectors. For `embed_image`, also remember the digest is
+over the URL String: if you replace bytes at the same URL (PUT-replace
+on S3 without renaming), null the digest field to force re-embed.
 
 ---
 
@@ -491,7 +590,8 @@ on every poll) rather than a `until index_ready?; sleep` loop.
 Key files:
 
 * `lib/parse/embeddings.rb` — registry, `Configuration`, `register`,
-  `provider`, `configure`.
+  `provider`, `configure`, `validate_image_url!`,
+  `trust_provider_url_fetch=`, `allowed_image_hosts=`.
 * `lib/parse/embeddings/provider.rb` — abstract base, `validate_response!`,
   `instrument_embed`, AS::N payload contract.
 * `lib/parse/embeddings/openai.rb` — OpenAI provider.
@@ -504,7 +604,8 @@ Key files:
   local-gateway client.
 * `lib/parse/embeddings/fixture.rb` — deterministic test provider.
 * `lib/parse/model/core/vector_searchable.rb` — `find_similar`.
-* `lib/parse/model/core/embed_managed.rb` — `embed` macro.
+* `lib/parse/model/core/embed_managed.rb` — `embed` and `embed_image`
+  macros, `EmbedDirective` (carries `modality:`, `allow_insecure:`).
 * `lib/parse/vector_search.rb` — low-level `Parse::VectorSearch.search`.
 * `lib/parse/atlas_search/index_manager.rb` — `IndexCatalog.create_index`,
   `find_vector_index`, `wait_for_ready`.

data/docs/client_sdk_guide.md

@@ -504,6 +504,13 @@ server-side guidance.
 
 ## 4. ACL — the row-level boundary
 
+> **For the full ACL + CLP reference**, including aggregate-query
+> enforcement asymmetry, Atlas Search, mongo-direct, role hierarchy
+> direction, `protectedFields` semantics including the `_User`
+> owner-exempt trap, and field-guard write protection, see
+> [`acl_clp_guide.md`](./acl_clp_guide.md). The sections below are
+> a client-mode quickstart.
+
 Parse Server enforces ACL on every read and write against a non-master
 caller. The SDK's job is to (a) thread the session token in so the server
 has someone to check against, and (b) compose ACLs correctly on the
@@ -707,7 +714,74 @@ Both `Parse.client.fetch_object` and `Parse::Query#results` strip the
 protected field; the SDK doesn't try to re-synthesize it from any
 cache. If you see it in your client-side result, your CLP is wrong.
 
-### 6.3 ACL still applies under CLP
+Reads are stripped today; the **write** response historically still
+echoed the value back in the create/update reply. Parse Server's
+`protectedFieldsSaveResponseExempt` option closes that — its default
+**will change to `false`** in a future version, which strips
+`protectedFields` from write responses too. Set
+`protectedFieldsSaveResponseExempt: false` in your server config to opt
+in early. The SDK needs no changes: `save` merges the response onto the
+object (it only overwrites fields the reply contains), so a stripped
+protected field keeps its locally-assigned value rather than being
+nulled out.
+
+### 6.3 `_Installation` is special — CLP can't override the hardcoded gates
+
+Parse Server hardcodes the access policy for `_Installation` at the REST
+layer. CLP on this class is a thin overlay on top of behavior that is
+already constrained, so `set_clp` (or a server-side CLP edit via the
+Dashboard) can only tighten the operations Parse Server lets you
+configure — it cannot loosen the ones the server pins to master-only.
+
+| Operation  | What's actually enforced                                                                 |
+|------------|------------------------------------------------------------------------------------------|
+| `find`     | **Master key only. Hardcoded.** CLP changes are ignored by the server.                  |
+| `delete`   | **Master key only. Hardcoded.** CLP changes are ignored by the server.                  |
+| `create`   | Open to anonymous clients — `X-Parse-Installation-Id` is the credential.                |
+| `update`   | Open when the request's `installationId` matches the record; else master key.           |
+| `get`      | CLP applies normally.                                                                   |
+| `count`    | CLP applies normally.                                                                   |
+| `addField` | CLP applies normally.                                                                   |
+
+Safe to tighten:
+
+* `get` → `requiresAuthentication: true` or master-only. SDKs don't
+  normally GET their own installation from the server; they cache
+  `currentInstallation` locally.
+* `count` → master-only. The push flow doesn't need it, and it removes a
+  small enumeration signal.
+* `addField` → master-only. Good hardening default for any class.
+* `protectedFields` → hide `deviceToken`, `GCMSenderId`, `pushType` from
+  non-master reads. These are write-only from the client's perspective
+  in normal SDK flows.
+
+Do NOT tighten:
+
+* `create` requiring authentication — breaks first-launch device
+  registration for users who haven't logged in yet. If your app pushes
+  to anonymous users, this kills it.
+* `update` requiring authentication — breaks silent device-token
+  refresh and channel subscribe/unsubscribe before login.
+* Pointer-based `readUserFields` / `writeUserFields` on `_Installation`
+  — a device has no stable owning user (it can outlive a session and
+  change users), so user-pointer ACLing is unreliable.
+* Anything on `find` / `delete` — the server ignores it.
+
+If your app genuinely requires login before any installation write, put
+the policy in a `beforeSave('_Installation')` Cloud Code trigger rather
+than in CLP:
+
+```js
+Parse.Cloud.beforeSave('_Installation', ({ user, master }) => {
+  if (!master && !user) throw 'login required';
+});
+```
+
+The trigger fires under master-key context and can inspect `request.user`
+directly without disturbing the anonymous registration handshake that
+the client SDKs depend on.
+
+### 6.4 ACL still applies under CLP
 
 CLP says "is this class operation allowed at all?". ACL says "given the
 operation is allowed, which rows does this caller see / touch?". An
@@ -715,6 +789,104 @@ authed user who passed the CLP gate still gets their result set filtered
 by ACL — if Alice writes a row with `acl.apply(alice.id, true, true)`
 only, Bob's query for it (under his own session) returns nothing.
 
+### 6.5 The other system classes — where CLP isn't the whole story
+
+`_Installation` (section 6.3) isn't unique. Several Parse Server system
+classes either ignore CLP entirely or layer it under hardcoded behavior.
+Treat this table as the authoritative answer for "what can I actually
+configure here?":
+
+| Class                                                                                            | Does CLP do anything?                                                                                              |
+|--------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
+| `_User`                                                                                          | Yes, but layered under hardcoded protections (password never returned, `authData` stripped from non-master finds, unauth update requires matching session token, email/username lowercasing, owner-exempt `protectedFields`). |
+| `_Role`                                                                                          | Yes, layered under role-name regex, relation validation, and hierarchy integrity checks.                            |
+| `_Installation`                                                                                  | Partial — only `get`, `count`, `addField`, and `protectedFields` are configurable; `find` and `delete` are master-only regardless of CLP. See section 6.3. |
+| `_Session`                                                                                       | Mostly redundant — non-master queries are silently rewritten to `{ user: <current user> }` (`RestQuery.js`), so a caller only ever sees their own sessions. `find` also requires a session token. |
+| `_JobStatus`, `_PushStatus`, `_Hooks`, `_GlobalConfig`, `_GraphQLConfig`, `_JobSchedule`, `_Audience`, `_Idempotency`, `_Join:*` (all relation join tables) | No — master-key-only at the REST layer (`SharedRest.js`). CLP changes are ignored.                                  |
+
+Practical consequences when you're building a client-mode app:
+
+* Don't query `_JobStatus`, `_PushStatus`, `_Audience`, `_JobSchedule`,
+  or any `_Join:*` table from the client — those calls require master
+  key. In the SDK, the corresponding model classes
+  (`Parse::JobStatus`, `Parse::PushStatus`, `Parse::Audience`,
+  `Parse::JobSchedule`) are server-side helpers. Auto-promote to a
+  master-key client or expose them through a Cloud Code function.
+* On `_Session`, you don't need ACLs or CLP to scope queries to the
+  caller — Parse Server already does that. You also can't grant a user
+  visibility into another user's sessions through CLP.
+* On `_User`, never assume CLP alone gates a flow. Password changes,
+  email verification, and session-token rotation have their own paths
+  in `RestWrite.js`; they fire even if your CLP looks restrictive.
+* On `_Role`, the role graph is validated server-side. CLP can gate
+  who can call create/update/delete, but the contents of the `roles`
+  and `users` relations are still checked for cycles and bad names.
+
+### 6.6 `_User` field visibility — master-only vs. self-only
+
+Two patterns come up constantly on `_User`:
+
+* **Master-only fields** — admin-side metadata that the user themselves
+  should not see. Examples: `my_opinion_of_them`, `risk_score`,
+  `moderation_notes`.
+* **Self-visible fields** — private profile data the user should see
+  on their own row, but nobody else. Examples: `favorite_color`,
+  `private_notes`, full `email`.
+
+Vanilla `protectedFields` doesn't express either cleanly on `_User`,
+because Parse Server's `protectedFieldsOwnerExempt` option (historical
+default **true**) silently exempts the owning user from every
+`protectedFields` rule on `_User`. So if you write `protect_fields "*",
+[:risk_score]`, the user still sees their own `risk_score`. (Parse
+Server's default for this option **is changing to `false`** in a future
+version; until your server adopts that default, set it explicitly as in
+step 1.) The fix has two moving parts on the server:
+
+1. Start Parse Server with `protectedFieldsOwnerExempt: false`.
+2. Add a self-pointer field on `_User` and populate it from a Cloud
+   Code trigger so each row points at itself:
+
+   ```js
+   Parse.Cloud.beforeSave(Parse.User, (req) => {
+     const u = req.object;
+     if (!u.get('self')) u.set('self', u);
+   });
+   ```
+
+With those in place, the SDK exposes a small DSL on `Parse::User`:
+
+```ruby
+class Parse::User
+  property :my_opinion_of_them, :string
+  property :favorite_color, :string
+
+  master_only_fields :my_opinion_of_them
+  self_visible_fields :favorite_color, via: :self   # name of the self-pointer
+end
+```
+
+That expands to:
+
+```ruby
+protect_fields "*",                ["myOpinionOfThem", "favoriteColor"]
+protect_fields "userField:self",   ["myOpinionOfThem"]
+```
+
+Resolution (recall: matching groups intersect):
+
+| Caller         | Matching groups            | Hidden (intersection)               | Visible       |
+|----------------|----------------------------|-------------------------------------|---------------|
+| Other user     | `*`                        | `myOpinionOfThem`, `favoriteColor`  | neither       |
+| The user itself| `*` ∩ `userField:self`     | `myOpinionOfThem` only              | `favoriteColor` |
+| Master key     | none (master bypasses)     | nothing                             | both          |
+
+If your code uses raw `protect_fields` on `_User` directly, the SDK
+emits a one-time advisory pointing at these helpers and reminding you
+to set `protectedFieldsOwnerExempt: false`. You can still use the raw
+form — the override calls through — but the warning is there because
+the default Parse Server setting will silently negate a lot of what
+the raw `protect_fields` calls look like they're doing.
+
 ---
 
 ## 7. Files
@@ -966,11 +1138,35 @@ server-side on every event before it goes out the WebSocket — Bob will
 not receive an event for an ACL-private row Alice creates, even if his
 subscription matches the `where` clause.
 
+> **Master-key authorization is per-CONNECTION, not per-subscription.**
+> Parse Server resolves master-key (ACL/CLP-bypass) authorization once,
+> from the connect frame; once set, EVERY subscription on that socket
+> bypasses ACL/CLP. The SDK therefore keeps connections **ACL-scoped by
+> default**: a configured `master_key` does NOT elevate the connection.
+> To build an admin (ACL-bypassing) connection — an event tap that sees
+> every row regardless of ACL — opt in explicitly:
+>
+> ```ruby
+> admin = Parse::LiveQuery::Client.new(
+>   url: "wss://parse.example.com/parse",
+>   application_id: "MY_APP_ID",
+>   master_key: ENV["PARSE_MASTER_KEY"],
+>   use_master_key: true,   # whole connection bypasses ACL/CLP; warns at connect
+> )
+> ```
+>
+> There is no per-subscription master key — `subscribe(use_master_key: true)`
+> on a scoped connection warns and stays ACL-scoped. For a process that
+> needs both scoped and admin streams, use two separate clients. Use
+> `client.admin_connection?` to check whether a connection is elevated.
+
 > **Configuration tip.** `Parse::LiveQuery::Client.new` reads
-> `master_key` from configuration if you omit it. Pass `master_key: nil`
-> **explicitly** in client builds — the SDK preserves a sentinel value
-> internally so it can tell "not provided" apart from "explicitly nil,"
-> and the latter is the only safe choice in a client context.
+> `master_key` from configuration if you omit it. Passing
+> `master_key: nil` **explicitly** in client builds is still good
+> hygiene (the SDK preserves a sentinel so it can tell "not provided"
+> apart from "explicitly nil"), but note that as of v5.1.0 a present
+> master key alone no longer elevates a LiveQuery connection — only
+> `use_master_key: true` does.
 
 ---
 

data/docs/usage_guide.md

@@ -394,6 +394,27 @@ Song.query(tags: tag).results          # songs containing this tag
 tag.songs.results                      # inverse query, if Tag declares has_many :songs, through: :relation
 ```
 
+### Heads up: Parse Server request-complexity limits
+
+Recent Parse Server versions add `requestComplexity` limits whose
+defaults are changing from "unlimited" (`-1`) to finite values in a
+future release: `includeDepth: 10`, `includeCount: 100`,
+`subqueryDepth: 10`, `queryDepth: 10`, and `batchRequestLimit: 100`.
+These cap how deep an `includes:` chain can nest, how many include
+paths a single query may carry, how deeply `matches_query` /
+`$inQuery` / `$select` subqueries nest, how deeply `$and` / `$or`
+conditions nest, and how many sub-requests a batch may contain.
+
+The SDK's defaults stay within these limits — most relevantly, the
+batch segment size is **50** (`Parse::BatchOperation#submit`), under the
+incoming `batchRequestLimit: 100`. The cases to watch are app-specific:
+very deep `includes: [{a: {b: {c: …}}}]` chains, queries with many
+distinct include paths, or deeply nested subqueries can start returning
+errors once the finite defaults land. If you hit one, restructure the
+query (split it, fetch pointers lazily, flatten the nesting) or raise
+the specific `requestComplexity.*` limit on your server. Set any of them
+to `-1` to opt out of that limit entirely.
+
 ## Atomic Operations
 
 Use atomic ops to avoid read-modify-write races on counters, sets, and