json-rest-api 2.2.0 → 2.3.0

package/docs/GUIDE/21-schema-and-migrations.md

@@ -457,7 +457,8 @@ Slot limits belong to the database: every server reads the slot columns that
 date, 5 JSON and 5 belongsTo slots. `ensureAnyApiSchema(knex, { slots })`, or the
 plugin's `slots` option when it ensures the schema, adds missing columns. Lower
 counts never drop columns. A server whose pool is exhausted rereads the layout
-once, so a migration that adds slots takes effect without a restart.
+once, so a migration that adds slots takes effect without a restart. Indexed
+slots have their own counts; see [AnyAPI indexed fields](#anyapi-indexed-fields).
 
 On PostgreSQL, adding a nullable column without a default does not rewrite the
 table. MySQL can add columns instantly from 8.0.29; earlier versions may copy the
@@ -498,6 +499,61 @@ lookups return null for absent metadata; operations requiring that relationship
 descriptor reject a missing descriptor. Schema errors and database failures
 are distinct from a valid relationship with no matching records.
 
+### AnyAPI indexed fields
+
+All AnyAPI resources share `any_records`, so plain slots have no indexes: a filter
+or sort on one reads every record of that tenant's resource. Indexed slots
+(`indexed_string_N`, `indexed_number_N`, `indexed_boolean_N`, `indexed_date_N`)
+each carry an index on `(tenant_id, resource, slot)`. A field declared
+`indexed: true` takes the first free indexed slot of its type:
+
+```js
+await api.addResource('contacts', {
+  schema: {
+    email: { type: 'email', indexed: true, search: true },
+    score: { type: 'integer', indexed: true },
+    notes: { type: 'string' }
+  }
+})
+```
+
+New tables get 5 string, 3 number and 2 date indexed slots, and no boolean ones.
+`ensureAnyApiSchema(knex, { indexedSlots: { string: 8, boolean: 2 } })`, or the
+plugin's `indexedSlots` option, adds more; JSON slots cannot be indexed. When a
+resource has no free indexed slot of a type, registration and `addKnexFields`
+fail with an error naming `indexedSlots`. The registry rereads the layout once,
+so a migration that adds slots takes effect without a restart. Every index adds
+work to each write to `any_records`, whatever the resource, so reserve indexed
+slots for fields that large resources filter or sort on.
+
+Indexed string slots are `VARCHAR(255)`, so MySQL can index them whole. A field
+whose `max` is larger cannot take one, and longer values fail validation with a
+`storage_range` violation on every database. Lengths count characters, not bytes.
+
+Slots never move on their own. A field keeps its slot when a declaration adds or
+drops `indexed`, and a server whose declaration marks a field indexed that is
+stored in a plain slot logs a warning. To index such a field, copy it into a new
+indexed field:
+
+```js
+await api.resources.contacts.migrateField({
+  from: 'phone',
+  to: { name: 'phoneKey', type: 'string', nullable: true, indexed: true, search: true }
+})
+```
+
+Belongs-to slots are always indexed by parent ID, and `any_links` by each side's
+record ID, so child lookups, includes and many-to-many reads stay inside one
+tenant's rows.
+
+`ensureAnyApiSchema` adds missing slot columns and indexes to existing tables too.
+MySQL builds indexes in place and PostgreSQL concurrently, so writes continue; on
+PostgreSQL one server builds at a time and the others start without waiting. A
+PostgreSQL pool with a single connection builds without `CONCURRENTLY`, which
+blocks writes. Either way the build reads the whole table, so for a large
+`any_records` run `ensureAnyApiSchema` from a migration before deploying. An
+interrupted concurrent build leaves an invalid index, which the next run rebuilds.
+
 ## Live Table Snapshots
 
 Use `introspectKnexTableSnapshot()` to inspect the physical table shape:

package/docs/GUIDE/30-backend-capabilities.md

@@ -67,7 +67,9 @@ AnyAPI field additions persist slot metadata and their definitions;
 and keeps the mapping of populated fields a declaration omits; changing a populated
 field's storage type needs an explicit data migration. Servers sharing a database
 see each other's additions after a reload, notification or schema check, and slot
-limits follow the columns the database has. Old canonical temporal storage needs
+limits follow the columns the database has. Plain slots have no indexes; fields
+declared `indexed: true` take [indexed slots](21-schema-and-migrations.md#anyapi-indexed-fields),
+and belongs-to slots are always indexed. Old canonical temporal storage needs
 the documented data migration, not just a package update. See
 [table helpers and migration limits](21-schema-and-migrations.md#anyapi-across-several-servers).
 

package/docs/GUIDE/33-migrating-to-v2.md

@@ -1485,6 +1485,17 @@ are, and omitted populated fields keep their data without being published.
 Fields added before this release that the declaration does not repeat are kept
 and published from their stored definitions.
 
+`ensureAnyApiSchema` also adds indexed slot columns (`indexed_string_N` and so on),
+indexes on `(tenant_id, resource, slot)` for them and for every belongs-to slot,
+and `(tenant_id, left_id)` and `(tenant_id, right_id)` indexes on `any_links`. On
+an existing table the plugin builds them at start-up without blocking writes, but
+the build reads the whole table: for a large one, run `ensureAnyApiSchema` in a
+migration before deploying. On AnyAPI, `indexed: true` now places fields. A new
+field declaring it takes an indexed slot, and registration fails when none is
+free, when the field is an `object` or `array`, or when a string field's `max`
+exceeds 255. Existing fields stay in their slots; see
+[AnyAPI indexed fields](21-schema-and-migrations.md#anyapi-indexed-fields).
+
 Explicit maps take effect during registration, before `createKnexTable()`.
 Repeating that method no longer re-registers the initial schema and discards
 additions. Keep the expanded declaration and complete map in app source for

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "json-rest-api",
-  "version": "2.2.0",
+  "version": "2.3.0",
   "main": "index.js",
   "types": "index.d.ts",
   "type": "module",

package/plugins/core/lib/anyapi/anyapi-registry.js

@@ -1,6 +1,6 @@
 import { createKnexTransaction, readCommittedTransaction } from '../../../../lib/knex-transaction.js'
 import { applyInsertReturning, getDatabaseCapabilities } from '../querying-writing/database-capabilities.js'
-import { DEFAULT_CANONICAL_CONFIG, TYPE_TO_POOL, ensureAnyApiSchema, readSlotPools } from './schema-utils.js'
+import { DEFAULT_CANONICAL_CONFIG, INDEXED_STRING_LENGTH, TYPE_TO_POOL, ensureAnyApiSchema, readSlotPools } from './schema-utils.js'
 import { toStoredJson } from './stored-definition.js'
 import { beginWriteTransaction, commitTransaction, endWriteOperation, getOperationDiagnosticContext, rollbackAfterError, wrapUnexpectedError, wrapWriteError } from '../../../../lib/error-context.js'
 import { createEnhancedLogger } from '../../../../lib/enhanced-logger.js'
@@ -46,17 +46,13 @@ const BELONGS_TO_ALIAS = (fieldName, fieldDef) => {
   return fieldName
 }
 
-const SLOT_PATTERNS = {
-  string: /^string_(\d+)$/,
-  number: /^number_(\d+)$/,
-  boolean: /^boolean_(\d+)$/,
-  date: /^date_(\d+)$/,
-  json: /^json_(\d+)$/,
-}
+const SIMPLE_SLOT_PATTERN = /^(indexed_)?(string|number|boolean|date|json)_(\d+)$/
 
 const BELONGS_TO_ID_PATTERN = /^rel_(\d+)_id$/
 
-const slotsExhausted = pool => Object.assign(new Error(`No available ${pool} slots remaining`), { slotsExhausted: true })
+const slotsExhausted = (pool, indexed) => Object.assign(new Error(indexed
+  ? `No available indexed ${pool} slots remaining; add them with ensureAnyApiSchema(knex, { indexedSlots: { ${pool}: count } })`
+  : `No available ${pool} slots remaining`), { slotsExhausted: true })
 
 const slotPoolFor = (fieldDef = {}) => {
   if (fieldDef.belongsTo) return 'belongsTo'
@@ -65,17 +61,18 @@ const slotPoolFor = (fieldDef = {}) => {
 
 const belongsToColumns = index => [`rel_${index}_id`, `rel_${index}_type`]
 
+const isIndexedSlot = column => column.startsWith('indexed_')
+
+const slotColumnsOf = (pools, pool, indexed) => (indexed ? pools.indexed?.[pool] : pools[pool]) || []
+
 const parseSimpleSlotColumn = (column, pools) => {
-  for (const [type, pattern] of Object.entries(SLOT_PATTERNS)) {
-    const match = pattern.exec(column)
-    if (match) {
-      if (!pools[type].includes(column)) {
-        throw new Error(`Canonical slot '${column}' is out of range`)
-      }
-      return { slotType: type, slotIndex: Number(match[1]) }
-    }
+  const match = SIMPLE_SLOT_PATTERN.exec(column)
+  if (!match) throw new Error(`Invalid canonical slot column '${column}'`)
+  const [, indexed, slotType, slotIndex] = match
+  if (!slotColumnsOf(pools, slotType, Boolean(indexed)).includes(column)) {
+    throw new Error(`Canonical slot '${column}' is out of range`)
   }
-  throw new Error(`Invalid canonical slot column '${column}'`)
+  return { slotType, slotIndex: Number(slotIndex) }
 }
 
 const parseBelongsToSlots = ({ idSlot, typeSlot }, pools) => {
@@ -148,9 +145,9 @@ export class AnyapiRegistry {
     return this.#shared.get(key)
   }
 
-  ensureSchema ({ slots } = {}) {
-    return this.#once(`schema:${JSON.stringify(slots ?? null)}`, async () => {
-      await ensureAnyApiSchema(this.knex, { slots })
+  ensureSchema ({ slots, indexedSlots } = {}) {
+    return this.#once(`schema:${JSON.stringify([slots ?? null, indexedSlots ?? null])}`, async () => {
+      await ensureAnyApiSchema(this.knex, { slots, indexedSlots })
       this.#shared.delete('slots')
     })
   }
@@ -757,9 +754,10 @@ export class AnyapiRegistry {
       const previous = previousDescriptor?.fields?.[entry.fieldName]
       if (entry.override || !previous || previous.slotType !== slotPoolFor(entry.fieldDef)) continue
       const columns = previous.slotType === 'belongsTo' ? belongsToColumns(previous.slotIndex) : [previous.slot]
+      // A field keeps its slot even when its `indexed` declaration changed.
       const available = previous.slotType === 'belongsTo'
         ? pools.belongsTo.includes(previous.slotIndex)
-        : pools[previous.slotType]?.includes(previous.slot)
+        : slotColumnsOf(pools, previous.slotType, isIndexedSlot(previous.slot)).includes(previous.slot)
       if (!available || columns.some(column => used.has(column))) continue
       columns.forEach(column => used.add(column))
       entry.override = previous.slotType === 'belongsTo' ? { idSlot: columns[0], typeSlot: columns[1] } : { slotColumn: previous.slot }
@@ -869,6 +867,8 @@ export class AnyapiRegistry {
         slot: row.slot_column,
         slotType: row.slot_type,
         slotIndex: row.slot_index,
+        // belongsTo slots are always indexed by parent id.
+        indexed: row.slot_type === 'belongsTo' || isIndexedSlot(row.slot_column),
         nullable: row.nullable === 1 || row.nullable === true,
         required: row.required === 1 || row.required === true,
         target: row.target_resource,
@@ -998,16 +998,29 @@ export class AnyapiRegistry {
           `Canonical slot '${slotColumn}' does not match expected type '${pool}' for field '${fieldDef.fieldName}'`
         )
       }
+      this.#assertFitsSlot(fieldDef, pool, slotColumn)
       used.add(slotColumn)
       return { slotType: pool, slotIndex, slotColumn }
     }
 
-    const slotColumn = pools[pool].find(column => !used.has(column))
+    const indexed = fieldDef.indexed === true
+    if (indexed && !pools.indexed?.[pool]) {
+      throw new Error(`Field '${fieldDef.fieldName}' is declared indexed, but AnyAPI ${pool} slots cannot be indexed`)
+    }
+    const columns = slotColumnsOf(pools, pool, indexed)
+    const slotColumn = columns.find(column => !used.has(column))
     if (!slotColumn) {
-      throw slotsExhausted(pool)
+      throw slotsExhausted(pool, indexed)
     }
+    this.#assertFitsSlot(fieldDef, pool, slotColumn)
     used.add(slotColumn)
-    return { slotType: pool, slotIndex: pools[pool].indexOf(slotColumn) + 1, slotColumn }
+    return { slotType: pool, slotIndex: columns.indexOf(slotColumn) + 1, slotColumn }
+  }
+
+  #assertFitsSlot (fieldDef, pool, slotColumn) {
+    if (pool === 'string' && isIndexedSlot(slotColumn) && Number(fieldDef.max) > INDEXED_STRING_LENGTH) {
+      throw new Error(`Field '${fieldDef.fieldName}' allows ${fieldDef.max} characters, but indexed string slots hold at most ${INDEXED_STRING_LENGTH}`)
+    }
   }
 
   #assignBelongsTo (fieldDef, used, canonicalOverride, pools) {

package/plugins/core/lib/anyapi/schema-utils.js

@@ -1,7 +1,7 @@
-import { hasKnexTableIndex } from '../dbIntrospection.js'
+import { hasKnexTableIndex, listKnexTableIndexes } from '../dbIntrospection.js'
 
-// Default layout of a new any_records table. The registry reads the actual
-// columns, so a migration that adds slots raises the limits for every server.
+// Default layout of any_records. The registry reads the actual columns, so a
+// migration that adds slots raises the limits for every server.
 export const SLOT_LIMITS = Object.freeze({
   string: 10,
   number: 10,
@@ -11,6 +11,18 @@ export const SLOT_LIMITS = Object.freeze({
   belongsTo: 5,
 })
 
+// Indexed slots carry a (tenant_id, resource, slot) index. Fields declared
+// `indexed: true` take them; every index costs each write to any_records.
+export const INDEXED_SLOT_LIMITS = Object.freeze({
+  string: 5,
+  number: 3,
+  boolean: 0,
+  date: 2,
+})
+
+// Indexed string slots are VARCHAR so MySQL can index them whole.
+export const INDEXED_STRING_LENGTH = 255
+
 const SLOT_COLUMNS = {
   string: index => [`string_${index}`],
   number: index => [`number_${index}`],
@@ -20,25 +32,37 @@ const SLOT_COLUMNS = {
   belongsTo: index => [`rel_${index}_id`, `rel_${index}_type`],
 }
 
+export const indexedSlotColumn = (pool, index) => `indexed_${pool}_${index}`
+
 const addSlotColumn = (table, column) => {
-  if (column.startsWith('number_')) table.double(column)
-  else if (column.startsWith('boolean_')) table.boolean(column)
-  else if (column.startsWith('date_')) table.dateTime(column, { precision: 6 })
-  else if (column.startsWith('rel_')) table.string(column)
+  const [, indexed, pool] = /^(indexed_)?([a-z]+)_/.exec(column)
+  if (pool === 'number') table.double(column)
+  else if (pool === 'boolean') table.boolean(column)
+  else if (pool === 'date') table.dateTime(column, { precision: 6 })
+  else if (pool === 'rel' || indexed) table.string(column, INDEXED_STRING_LENGTH)
   else table.text(column)
 }
 
-const resolveSlotCounts = (slots = {}) => {
+const resolveSlotCounts = (slots = {}, limits = SLOT_LIMITS, { label = 'slot', minimum = 1 } = {}) => {
   if (!slots || typeof slots !== 'object' || Array.isArray(slots)) {
-    throw new Error('AnyAPI slots must be an object of slot counts')
+    throw new Error(`AnyAPI ${label}s must be an object of slot counts`)
   }
   for (const [pool, count] of Object.entries(slots)) {
-    if (!Object.hasOwn(SLOT_LIMITS, pool)) throw new Error(`Unknown AnyAPI slot pool '${pool}'`)
-    if (!Number.isInteger(count) || count < 1) throw new Error(`AnyAPI ${pool} slot count must be a positive integer`)
+    if (!Object.hasOwn(limits, pool)) throw new Error(`Unknown AnyAPI ${label} pool '${pool}'`)
+    if (!Number.isInteger(count) || count < minimum) {
+      throw new Error(`AnyAPI ${pool} ${label} count must be ${minimum ? 'a positive' : 'a non-negative'} integer`)
+    }
   }
-  return { ...SLOT_LIMITS, ...slots }
+  return { ...limits, ...slots }
 }
 
+const layoutColumns = (counts, indexedCounts) => [
+  ...Object.entries(counts).flatMap(([pool, count]) => Array.from({ length: count }, (_, index) => SLOT_COLUMNS[pool](index + 1)).flat()),
+  ...Object.entries(indexedCounts).flatMap(([pool, count]) => Array.from({ length: count }, (_, index) => indexedSlotColumn(pool, index + 1)))
+]
+
+const slotIndexName = column => `any_records_${column}_idx`
+
 export const DEFAULT_CANONICAL_CONFIG = {
   scopeName: 'anyapi_records',
   tableName: 'any_records',
@@ -53,52 +77,121 @@ export const DEFAULT_CANONICAL_CONFIG = {
   rightResourceColumn: 'right_resource',
 }
 
-const ensureTable = async (knex, tableName, tableBuilder) => {
-  const exists = await knex.schema.hasTable(tableName)
-  if (!exists) {
-    await knex.schema.createTable(tableName, tableBuilder)
+// Several servers may start at once: DDL another one completed first is success.
+const tolerateConcurrentDdl = async (run, done) => {
+  try {
+    await run()
+  } catch (error) {
+    if (!await done().catch(() => false)) throw error
   }
 }
 
+// Returns whether the table was missing, so its indexes can be built offline.
+const ensureTable = async (knex, tableName, tableBuilder) => {
+  if (await knex.schema.hasTable(tableName)) return false
+  await tolerateConcurrentDdl(() => knex.schema.createTable(tableName, tableBuilder), () => knex.schema.hasTable(tableName))
+  return true
+}
+
 const ensureColumn = async (knex, tableName, columnName, columnBuilder) => {
   const exists = await knex.schema.hasTable(tableName)
   if (!exists) return
 
-  const hasColumn = await knex.schema.hasColumn(tableName, columnName)
-  if (!hasColumn) {
-    await knex.schema.table(tableName, columnBuilder)
+  const hasColumn = () => knex.schema.hasColumn(tableName, columnName)
+  if (!await hasColumn()) {
+    await tolerateConcurrentDdl(() => knex.schema.table(tableName, columnBuilder), hasColumn)
   }
 }
 
 const ensureUniqueIndex = async (knex, tableName, columns, indexName) => {
   const exists = await knex.schema.hasTable(tableName)
   if (!exists) return
-  const hasIndex = await hasKnexTableIndex(knex, tableName, indexName)
-  if (!hasIndex) {
-    await knex.schema.table(tableName, (table) => {
+  const hasIndex = () => hasKnexTableIndex(knex, tableName, indexName)
+  if (!await hasIndex()) {
+    await tolerateConcurrentDdl(() => knex.schema.table(tableName, (table) => {
       table.unique(columns, indexName)
-    })
+    }), hasIndex)
+  }
+}
+
+/**
+ * Create missing secondary indexes. On an existing table they are built without
+ * blocking writes: in place on MySQL, and concurrently on PostgreSQL unless the
+ * pool has a single connection. Either way the build reads the whole table, so
+ * on a big table run it from a migration before the deploy.
+ */
+const ensureIndexes = async (knex, tableName, indexes, { online }) => {
+  const client = knex.client.config.client
+  const pg = ['pg', 'postgresql'].includes(client)
+  const mysql = ['mysql', 'mysql2'].includes(client)
+  // The build lock needs a connection of its own beside the builds.
+  const concurrently = online && pg && !knex.isTransaction && (knex.client.pool?.max ?? 0) > 1
+  const missing = async () => {
+    const existing = await listKnexTableIndexes(knex, tableName)
+    return indexes.filter(({ name }) => !existing.get(name)?.valid && !existing.get(name)?.building)
+      .map(index => ({ ...index, invalid: existing.has(index.name) }))
+  }
+  const build = async () => {
+    for (const { name, columns, invalid } of await missing()) {
+      const list = columns.map(() => '??').join(', ')
+      if (invalid) await knex.raw(`drop index ${concurrently ? 'concurrently ' : ''}if exists ??`, [name])
+      if (mysql) {
+        await tolerateConcurrentDdl(
+          () => knex.raw(`alter table ?? add index ?? (${list})${online ? ', algorithm=inplace, lock=none' : ''}`, [tableName, name, ...columns]),
+          () => hasKnexTableIndex(knex, tableName, name)
+        )
+      } else {
+        await tolerateConcurrentDdl(
+          () => knex.raw(`create index ${concurrently ? 'concurrently ' : ''}if not exists ?? on ?? (${list})`, [name, tableName, ...columns]),
+          async () => (await listKnexTableIndexes(knex, tableName)).has(name)
+        )
+      }
+    }
+  }
+  if (!(await missing()).length) return
+  if (concurrently) await withPgIndexBuildLock(knex, tableName, build)
+  else await build()
+}
+
+// Concurrent builds on one table deadlock each other, so one server builds at a
+// time. The lock lives on an idle session, which concurrent builds never wait
+// for. A server that finds it taken carries on: the holder builds the same indexes.
+const withPgIndexBuildLock = async (knex, tableName, build) => {
+  const connection = await knex.client.acquireConnection()
+  const lock = name => knex.select(knex.raw(`${name}(hashtext(?)) as result`, [`json-rest-api:indexes:${tableName}`]))
+    .connection(connection).first()
+  try {
+    if (!(await lock('pg_try_advisory_lock')).result) return
+    try {
+      await build()
+    } finally {
+      await lock('pg_advisory_unlock')
+    }
+  } finally {
+    await knex.client.releaseConnection(connection)
   }
 }
 
 /**
  * Create or upgrade the shared canonical tables. Run it from a migration, or let
- * the plugin run it at start-up. `slots` only adds missing slot columns; lowering
- * a count never drops columns that may hold data.
+ * the plugin run it at start-up. Counts only add missing slot columns, never
+ * drop them; any_records always has at least the default layout.
  * @param {import('knex').Knex} knex
- * @param {{ slots?: Partial<Record<keyof typeof SLOT_LIMITS, number>> }} [options]
+ * @param {{
+ *   slots?: Partial<Record<keyof typeof SLOT_LIMITS, number>>,
+ *   indexedSlots?: Partial<Record<keyof typeof INDEXED_SLOT_LIMITS, number>>
+ * }} [options]
  */
-export const ensureAnyApiSchema = async (knex, { slots } = {}) => {
+export const ensureAnyApiSchema = async (knex, { slots, indexedSlots } = {}) => {
   const counts = resolveSlotCounts(slots)
-  await ensureTable(knex, 'any_records', (table) => {
+  const indexedCounts = resolveSlotCounts(indexedSlots, INDEXED_SLOT_LIMITS, { label: 'indexed slot', minimum: 0 })
+  const created = await ensureTable(knex, 'any_records', (table) => {
     table.increments('id').primary()
     table.string('logical_id')
     table.string('tenant_id').notNullable()
     table.string('resource').notNullable()
 
-    for (const [pool, count] of Object.entries(counts)) {
-      for (let index = 1; index <= count; index += 1) SLOT_COLUMNS[pool](index).forEach(column => addSlotColumn(table, column))
-    }
+    layoutColumns(counts, indexedCounts).forEach(column => addSlotColumn(table, column))
 
     table.dateTime('created_at').defaultTo(knex.fn.now()).notNullable()
     table.dateTime('updated_at').defaultTo(knex.fn.now()).notNullable()
@@ -109,14 +202,18 @@ export const ensureAnyApiSchema = async (knex, { slots } = {}) => {
     table.index(['resource'])
   })
 
-  if (slots) {
-    const existing = new Set(Object.keys(await knex('any_records').columnInfo()))
-    const missing = Object.entries(counts).flatMap(([pool, count]) => Array.from({ length: count }, (_, index) => SLOT_COLUMNS[pool](index + 1)))
-      .flat()
-      .filter(column => !existing.has(column))
-    if (missing.length) {
-      await knex.schema.alterTable('any_records', (table) => missing.forEach(column => addSlotColumn(table, column)))
-    }
+  const readColumns = async () => new Set(Object.keys(await knex('any_records').columnInfo()))
+  let recordColumns = await readColumns()
+  const missing = layoutColumns(counts, indexedCounts).filter(column => !recordColumns.has(column))
+  if (missing.length) {
+    await tolerateConcurrentDdl(
+      () => knex.schema.alterTable('any_records', (table) => missing.forEach(column => addSlotColumn(table, column))),
+      async () => {
+        const columns = await readColumns()
+        return missing.every(column => columns.has(column))
+      }
+    )
+    recordColumns = await readColumns()
   }
 
   const client = knex.client.config.client
@@ -125,7 +222,7 @@ export const ensureAnyApiSchema = async (knex, { slots } = {}) => {
       .select({ column: 'column_name', precision: 'datetime_precision' })
       .where('table_schema', knex.raw(client === 'pg' || client === 'postgresql' ? 'current_schema()' : 'DATABASE()'))
       .where('table_name', 'any_records'))
-      .filter(column => /^date_\d+$/.test(column.column))
+      .filter(column => /^(indexed_)?date_\d+$/.test(column.column))
     if (!columns.length || columns.some(column => Number(column.precision) !== 6)) {
       throw new Error('AnyAPI temporal storage migration required: every any_records date slot must use timestamp/datetime precision 6. See 33-migrating-to-v2.md.')
     }
@@ -151,7 +248,7 @@ export const ensureAnyApiSchema = async (knex, { slots } = {}) => {
       .update({ logical_id: String(row.id) })
   }
 
-  await ensureTable(knex, 'any_links', (table) => {
+  const linksCreated = await ensureTable(knex, 'any_links', (table) => {
     table.increments('id').primary()
     table.string('tenant_id').notNullable()
     table.string('relationship').notNullable()
@@ -172,6 +269,17 @@ export const ensureAnyApiSchema = async (knex, { slots } = {}) => {
     table.index(['inverse_relationship'])
   })
 
+  // Filters on indexed fields, child lookups by parent id and link lookups by
+  // record id all stay inside one tenant's rows.
+  const recordIndexes = [...recordColumns]
+    .filter(column => /^(indexed_[a-z]+_\d+|rel_\d+_id)$/.test(column))
+    .map(column => ({ name: slotIndexName(column), columns: ['tenant_id', 'resource', column] }))
+  await ensureIndexes(knex, 'any_records', recordIndexes, { online: !created })
+  await ensureIndexes(knex, 'any_links', [
+    { name: 'any_links_tenant_left_id_idx', columns: ['tenant_id', 'left_id'] },
+    { name: 'any_links_tenant_right_id_idx', columns: ['tenant_id', 'right_id'] }
+  ], { online: !linksCreated })
+
   await ensureTable(knex, 'any_resource_configs', (table) => {
     table.increments('id').primary()
     table.string('tenant_id').notNullable()
@@ -249,23 +357,35 @@ export const ensureAnyApiSchema = async (knex, { slots } = {}) => {
   })
 }
 
-const buildSlotPools = (hasColumns) => Object.fromEntries(Object.keys(SLOT_LIMITS).map(pool => {
-  const indexes = []
-  for (let index = 1; hasColumns(pool, index); index += 1) indexes.push(index)
+// The default unindexed layout, for migrations written against it.
+export const SLOT_POOLS = Object.fromEntries(Object.entries(SLOT_LIMITS).map(([pool, count]) => {
+  const indexes = Array.from({ length: count }, (_, index) => index + 1)
   return [pool, pool === 'belongsTo' ? indexes : indexes.map(index => SLOT_COLUMNS[pool](index)[0])]
 }))
 
-export const SLOT_POOLS = buildSlotPools((pool, index) => index <= SLOT_LIMITS[pool])
-
 /**
  * Read the slot columns any_records actually has. Slots are numbered from 1
- * without gaps; a pool ends at its first missing column.
+ * without gaps; a pool ends at its first missing column. `indexed` lists the
+ * indexed columns of each pool that has them.
  * @param {import('knex').Knex | import('knex').Knex.Transaction} db
  */
 export const readSlotPools = async (db) => {
   const columns = new Set(Object.keys(await db('any_records').columnInfo()))
   if (!columns.size) throw new Error('AnyAPI storage tables are missing; run ensureAnyApiSchema in a migration')
-  return buildSlotPools((pool, index) => SLOT_COLUMNS[pool](index).every(column => columns.has(column)))
+  const run = columnsAt => {
+    const indexes = []
+    for (let index = 1; columnsAt(index).every(column => columns.has(column)); index += 1) indexes.push(index)
+    return indexes
+  }
+  return {
+    ...Object.fromEntries(Object.keys(SLOT_LIMITS).map(pool => {
+      const indexes = run(SLOT_COLUMNS[pool])
+      return [pool, pool === 'belongsTo' ? indexes : indexes.map(index => SLOT_COLUMNS[pool](index)[0])]
+    })),
+    indexed: Object.fromEntries(Object.keys(INDEXED_SLOT_LIMITS).map(pool => [
+      pool, run(index => [indexedSlotColumn(pool, index)]).map(index => indexedSlotColumn(pool, index))
+    ]))
+  }
 }
 
 export const TYPE_TO_POOL = new Map([

package/plugins/core/lib/dbIntrospection.js

@@ -1237,6 +1237,34 @@ export async function hasKnexTableIndex (knex, tableName, indexName) {
   throw new Error(`Unsupported knex client '${client}' for hasKnexTableIndex`)
 }
 
+/**
+ * Index names of one table. On PostgreSQL an interrupted concurrent build leaves
+ * an invalid index; `building` tells it from one another session is building.
+ * @returns {Promise<Map<string, { valid: boolean, building: boolean }>>}
+ */
+export async function listKnexTableIndexes (knex, tableName) {
+  const client = knex.client.config.client
+  let rows
+  if (['sqlite3', 'better-sqlite3'].includes(client)) {
+    rows = normalizeRows(await knex.raw('PRAGMA index_list(??)', [tableName])).map(row => ({ name: row.name }))
+  } else if (['mysql', 'mysql2'].includes(client)) {
+    rows = (await knex('information_schema.statistics').distinct({ name: 'INDEX_NAME' })
+      .where({ TABLE_SCHEMA: knex.raw('DATABASE()'), TABLE_NAME: tableName }))
+  } else if (['pg', 'postgresql'].includes(client)) {
+    rows = normalizeRows(await knex.raw(`
+      select c.relname as name, i.indisvalid as valid,
+        exists (select 1 from pg_stat_progress_create_index p where p.index_relid = i.indexrelid) as building
+      from pg_index i
+      join pg_class c on c.oid = i.indexrelid
+      join pg_class t on t.oid = i.indrelid
+      join pg_namespace n on n.oid = t.relnamespace
+      where t.relname = ? and n.nspname = current_schema()`, [tableName]))
+  } else {
+    throw new Error(`Unsupported knex client '${client}' for listKnexTableIndexes`)
+  }
+  return new Map(rows.map(row => [row.name, { valid: row.valid !== false, building: row.building === true }]))
+}
+
 // Column-only ALTER must not impose the resource snapshot's ID or index restrictions.
 export async function introspectKnexColumnConstraints (knex, tableName) {
   requireKnexRaw(knex)

package/plugins/core/lib/querying/knex-query-helpers.js

@@ -7,6 +7,8 @@ import { resolveSortField } from './query-field-sort-helpers.js'
 import { databaseIdentityExpression } from '../querying-writing/database-value-normalizers.js'
 
 const referenceVisibility = Symbol('referenceVisibility')
+// SQL never matches null against a non-null operand with these.
+const NULL_REJECTING_OPERATORS = new Set(['=', '==', '!=', '<>', '<', '<=', '>', '>=', 'in', 'between', 'like', 'contains', 'startswith', 'endswith'])
 
 async function buildSearchJoinQuery ({ scopeName, scopes, context, adapterUtils, db }) {
   const scope = scopes[scopeName]
@@ -135,12 +137,20 @@ export function applyWhereForOperator ({ builder, columnRef, operator, value, kn
       if (selections.length === 0) this.whereRaw('1 = 0')
       for (const selection of selections) this.orWhereExists(selection)
     }
+    const visibleMatch = function () {
+      this.where(visible)
+      applyWhereForOperator({ builder: this, columnRef: columnRef.column, operator, value, knex })
+    }
+    // A hidden reference reads as null. When null cannot match, leave that branch
+    // out: an OR with it keeps planners from using an index on the column.
+    const values = Array.isArray(value) ? value : [value]
+    if (NULL_REJECTING_OPERATORS.has(String(operator || '=').toLowerCase()) && values.length && values.every(item => item != null)) {
+      builder[method](visibleMatch)
+      return
+    }
     // Compare the column itself to preserve driver affinity and collation.
     builder[method](function () {
-      this.where(function () {
-        this.where(visible)
-        applyWhereForOperator({ builder: this, columnRef: columnRef.column, operator, value, knex })
-      }).orWhere(function () {
+      this.where(visibleMatch).orWhere(function () {
         this.whereNot(visible)
         applyWhereForOperator({ builder: this, columnRef: knex.raw('null'), operator, value, knex })
       })

package/plugins/core/lib/storage/canonical-storage-mapping.js

@@ -1,6 +1,8 @@
 // @ts-check
 import { serializeFieldValueForStorage } from './storage-mapping.js'
 import { getPolymorphicLinkage } from '../querying-writing/relationship-contracts.js'
+import { INDEXED_STRING_LENGTH } from '../anyapi/schema-utils.js'
+import { RestApiValidationError } from '../../../../lib/rest-api-errors.js'
 
 /** @import { CanonicalDescriptor, StorageRow, StorageSchemaInfo, StorageWriteOptions } from './storage-types.js' */
 
@@ -34,6 +36,15 @@ export const translateCanonicalAttributesForStorage = (attributes = {}, descript
         textStorage: slot.slotType === 'string'
       })
       row[slot.slot] = definition?.type === 'id' && serialized != null ? String(serialized) : serialized
+      const stored = row[slot.slot]
+      // Databases count code points; one surrogate pair is one character.
+      if (slot.indexed && typeof stored === 'string' && stored.length > INDEXED_STRING_LENGTH && [...stored].length > INDEXED_STRING_LENGTH) {
+        const field = `data.attributes.${fieldName}`
+        throw new RestApiValidationError(`Field '${fieldName}' holds at most ${INDEXED_STRING_LENGTH} characters in its indexed slot.`, {
+          fields: [field],
+          violations: [{ field, rule: 'storage_range', message: `Indexed string slots hold at most ${INDEXED_STRING_LENGTH} characters.` }]
+        })
+      }
     }
 
     if (slot.slotType === 'belongsTo') {

package/plugins/core/lib/storage/storage-types.d.ts

@@ -79,6 +79,8 @@ export interface StorageInfo {
 export interface CanonicalFieldSlot {
   slot: string
   slotType: string
+  // The slot has a (tenant_id, resource, slot) index; belongsTo slots always do.
+  indexed?: boolean
   alias?: string
 }
 

package/plugins/core/rest-api-anyapi-knex-plugin.js

@@ -39,7 +39,7 @@ export const RestApiAnyapiKnexPlugin = {
     const options = pluginOptions || {}
     const knex = options.knex
     const tenantId = options.tenantId || DEFAULT_TENANT
-    const { ensureSchema = true, ownsKnex = true, onSchemaChanged, schemaCheckInterval, slots } = options
+    const { ensureSchema = true, ownsKnex = true, onSchemaChanged, schemaCheckInterval, slots, indexedSlots } = options
 
     if (!knex) {
       throw new Error('AnyapiKnexPlugin requires a knex instance')
@@ -55,7 +55,7 @@ export const RestApiAnyapiKnexPlugin = {
     }
 
     const registry = options.registry || new AnyapiRegistry({ knex, log })
-    if (ensureSchema) await registry.ensureSchema({ slots })
+    if (ensureSchema) await registry.ensureSchema({ slots, indexedSlots })
 
     api.anyapi = api.anyapi || {}
     api.anyapi.registry = registry
@@ -134,6 +134,15 @@ export const RestApiAnyapiKnexPlugin = {
           tenant: tenantId, scopeName, fields: hidden, filters: hiddenFilters
         })
       }
+      // Fields never move on their own; migrateField copies one into an indexed slot.
+      const unindexed = Object.entries(scopeOptions.schema || {})
+        .filter(([name, definition]) => definition?.indexed === true && descriptor.fields[name]?.indexed === false)
+        .map(([name]) => name)
+      if (unindexed.length) {
+        await log.warn(`AnyAPI resource '${scopeName}' declares indexed fields stored in unindexed slots`, {
+          tenant: tenantId, scopeName, fields: unindexed
+        })
+      }
       const signature = optionsSignature(scopeOptions)
       let schemaInfo = scope.vars.schemaInfo
       if (compiled?.signature === signature) schemaInfo = compiled.schemaInfo
@@ -330,7 +339,7 @@ export const RestApiAnyapiKnexPlugin = {
     })
 
     addResourceMethod('createKnexTable', async ({ scopeName }) => {
-      if (ensureSchema) await ensureAnyApiSchema(knex, { slots })
+      if (ensureSchema) await ensureAnyApiSchema(knex, { slots, indexedSlots })
       const descriptor = await registry.getDescriptor(tenantId, scopeName)
       if (!descriptor) throw new Error(`Descriptor not found for resource '${scopeName}'`)
       await publish(scopeName, descriptor)
@@ -453,14 +462,21 @@ export const RestApiAnyapiKnexPlugin = {
           lastId = row.id
           const value = normalizeStoredFieldValue(row[source], fromDefinition, { fieldName: from, resourceType: descriptor.resource })
           const next = transform ? await transform(value, { id: String(row[idColumn]) }) : value
+          const rejected = reason => {
+            const message = `Record '${row[idColumn]}' cannot migrate '${from}' to '${to}': ${reason}`
+            return new RestApiValidationError(message, { fields: [to], violations: [{ field: to, rule: 'field_migration', message }] })
+          }
           const { validatedObject, errors } = await schemaInfo.schemaInstance.patch({ [to]: next ?? null })
-          if (Object.keys(errors).length) {
-            const message = `Record '${row[idColumn]}' cannot migrate '${from}' to '${to}': ${Object.values(errors)[0].message}`
-            throw new RestApiValidationError(message, { fields: [to], violations: [{ field: to, rule: 'field_migration', message }] })
+          if (Object.keys(errors).length) throw rejected(Object.values(errors)[0].message)
+          let stored
+          try {
+            stored = translateCanonicalAttributesForStorage({ [to]: validatedObject[to] ?? null }, descriptor, {
+              schemaInfo, databaseClient: knex.client.config.client
+            })
+          } catch (error) {
+            if (error?.type === 'rest_api_validation') throw rejected(error.message)
+            throw error
           }
-          const stored = translateCanonicalAttributesForStorage({ [to]: validatedObject[to] ?? null }, descriptor, {
-            schemaInfo, databaseClient: knex.client.config.client
-          })
           await transaction(canonical.tableName).where('id', row.id).update({ [target]: stored[target] ?? null })
         }
       }

package/types/anyapi.d.ts

@@ -2,6 +2,15 @@ import type { Knex } from 'knex'
 
 export type AnyapiSlotPool = 'string' | 'number' | 'boolean' | 'date' | 'json' | 'belongsTo'
 export type AnyapiSlotCounts = Partial<Record<AnyapiSlotPool, number>>
+// Pools whose slots can carry a (tenant_id, resource, slot) index.
+export type AnyapiIndexedSlotPool = 'string' | 'number' | 'boolean' | 'date'
+export type AnyapiIndexedSlotCounts = Partial<Record<AnyapiIndexedSlotPool, number>>
+
+export interface AnyapiSchemaOptions {
+  slots?: AnyapiSlotCounts
+  // Fields declared `indexed: true` take these slots.
+  indexedSlots?: AnyapiIndexedSlotCounts
+}
 
 export interface AnyapiSchemaChange {
   tenant: string
@@ -21,7 +30,7 @@ export class AnyapiRegistry {
   constructor(options: AnyapiRegistryOptions)
   readonly knex: Knex
   readonly cacheSize: number
-  ensureSchema(options?: { slots?: AnyapiSlotCounts }): Promise<void>
+  ensureSchema(options?: AnyapiSchemaOptions): Promise<void>
   getSchemaVersion(tenant: string, resource: string, options?: { transaction?: Knex.Transaction }): Promise<number | null>
   // Retired fields keep their slots and values and reject new writes.
   retireFields(params: { tenant: string; resource: string; fields: string[] }, options?: { transaction?: Knex.Transaction }): Promise<Record<string, unknown>>
@@ -29,4 +38,4 @@ export class AnyapiRegistry {
 }
 
 /** Create or upgrade the canonical tables; counts only ever add slot columns. */
-export function ensureAnyApiSchema(knex: Knex, options?: { slots?: AnyapiSlotCounts }): Promise<void>
+export function ensureAnyApiSchema(knex: Knex, options?: AnyapiSchemaOptions): Promise<void>

package/types/plugin-options.d.ts

@@ -1,7 +1,7 @@
 import type { DirectResourceId, ResourceFormat, WriteReturning } from './representations.js'
 import type { RemovedResourceOptions } from './resource-methods.js'
 import type { Knex } from 'knex'
-import type { AnyapiRegistry, AnyapiSchemaChange, AnyapiSlotCounts } from './anyapi.js'
+import type { AnyapiIndexedSlotCounts, AnyapiRegistry, AnyapiSchemaChange, AnyapiSlotCounts } from './anyapi.js'
 
 export type RestApiPluginOptions = RemovedResourceOptions & {
   format?: ResourceFormat
@@ -50,6 +50,7 @@ export interface RestApiAnyapiKnexPluginOptions extends RestApiKnexPluginOptions
   // false leaves a shared Knex instance open when this API is released.
   ownsKnex?: boolean
   slots?: AnyapiSlotCounts
+  indexedSlots?: AnyapiIndexedSlotCounts
   // Called after a committed schema change, for example to notify other servers.
   onSchemaChanged?: (change: AnyapiSchemaChange) => void | Promise<void>
   // Milliseconds between schema version checks; 0 checks on every request.