@proofkit/fmodata 0.1.0-alpha.4 → 0.1.0-alpha.7

package/README.md

@@ -2,11 +2,14 @@
 
 A strongly-typed FileMaker OData API client.
 
-⚠️ WARNING: This library is in "alpha" status. The API is subject to change. Feedback is welcome on the [community forum](https://community.ottomatic.cloud/c/proofkit/13) or on [GitHub](https://github.com/proofgeist/proofkit/issues).
+⚠️ WARNING: This library is in "alpha" status. It's still in active development and the API is subject to change. Feedback is welcome on the [community forum](https://community.ottomatic.cloud/c/proofkit/13) or on [GitHub](https://github.com/proofgeist/proofkit/issues).
 
 Roadmap:
 
-- [ ] Batch operations
+- [ ] Crossjoin support
+- [x] Batch operations
+  - [ ] Automatically chunk requests into smaller batches (e.g. max 512 inserts per batch)
+- [x] Schema updates (add/update tables and fields)
 - [ ] Proper docs at proofkit.dev
 - [ ] @proofkit/typegen integration
 
@@ -78,7 +81,7 @@ if (data) {
 
 ## Core Concepts
 
-This library relies heavily on the builder pattern for defining your queries and operations. Most operations require a final call to `execute()` to send the request to the server. The builder pattern is designed to support batch operations in the future, allowing you to execute multiple operations in a single request as supported by the FileMaker OData API. **Note:** Batch operations are not yet supported but are planned before the production release. It's also helpful for testing the library, as you can call `getQueryString()` to get the OData query string without executing the request.
+This library relies heavily on the builder pattern for defining your queries and operations. Most operations require a final call to `execute()` to send the request to the server. The builder pattern allows you to build complex queries and also supports batch operations, allowing you to execute multiple operations in a single request as supported by the FileMaker OData API. It's also helpful for testing the library, as you can call `getQueryString()` to get the OData query string without executing the request.
 
 As such, there are layers to the library to help you build your queries and operations.
 
@@ -140,9 +143,9 @@ const contactsBase = new BaseTable({
     phone: z.string().optional(),
     createdAt: z.string(),
   },
-  idField: "id", // The primary key field
-  insertRequired: ["name", "email"], // optional: fields that are required on insert
-  updateRequired: ["email"], // optional: fields that are required on update
+  idField: "id", // The primary key field (automatically read-only)
+  required: ["phone"], // optional: additional required fields for insert (beyond auto-inferred)
+  readOnly: ["createdAt"], // optional: fields excluded from insert/update
 });
 ```
 
@@ -524,27 +527,30 @@ if (result.data) {
 }
 ```
 
-If you specify `insertRequired` fields in your base table, those fields become required:
+Fields are automatically required for insert if their validator doesn't allow `null` or `undefined`. You can specify additional required fields:
 
 ```typescript
 const usersBase = new BaseTable({
   schema: {
-    id: z.string(),
-    username: z.string(),
-    email: z.string(),
-    createdAt: z.string().optional(),
+    id: z.string(), // Auto-required (not nullable), but excluded from insert (idField)
+    username: z.string(), // Auto-required (not nullable)
+    email: z.string(), // Auto-required (not nullable)
+    phone: z.string().nullable(), // Optional by default
+    createdAt: z.string(), // Auto-required, but excluded (readOnly)
   },
-  idField: "id",
-  insertRequired: ["username", "email"], // These fields are required on insert
+  idField: "id", // Automatically excluded from insert/update
+  required: ["phone"], // Make phone required for inserts despite being nullable
+  readOnly: ["createdAt"], // Exclude from insert/update operations
 });
 
-// TypeScript will enforce that username and email are provided
+// TypeScript enforces: username, email, and phone are required
+// TypeScript excludes: id and createdAt cannot be provided
 const result = await db
   .from("users")
   .insert({
     username: "johndoe",
     email: "john@example.com",
-    // createdAt is optional
+    phone: "+1234567890", // Required because specified in 'required' array
   })
   .execute();
 ```
@@ -799,6 +805,334 @@ console.log(result.result.recordId);
 
 **Note:** OData doesn't support script names with special characters (e.g., `@`, `&`, `/`) or script names beginning with a number. TypeScript will catch these at compile time.
 
+## Batch Operations
+
+Batch operations allow you to execute multiple queries and operations together in a single request. All operations in a batch are executed atomically - they all succeed or all fail together. This is both more efficient (fewer network round-trips) and ensures data consistency across related operations.
+
+### Basic Batch with Multiple Queries
+
+Execute multiple read operations in a single batch:
+
+```typescript
+// Create query builders
+const contactsQuery = db.from("contacts").list().top(5);
+const usersQuery = db.from("users").list().top(5);
+
+// Execute both queries in a single batch
+const result = await db.batch([contactsQuery, usersQuery]).execute();
+
+if (result.data) {
+  // Result is a tuple matching the input builders
+  const [contacts, users] = result.data;
+
+  console.log("Contacts:", contacts);
+  console.log("Users:", users);
+}
+```
+
+### Mixed Operations (Reads and Writes)
+
+Combine queries, inserts, updates, and deletes in a single batch:
+
+```typescript
+// Mix different operation types
+const listQuery = db.from("contacts").list().top(10);
+const insertOp = db.from("contacts").insert({
+  name: "John Doe",
+  email: "john@example.com",
+});
+const updateOp = db.from("users").update({ active: true }).byId("user-123");
+
+// All operations execute atomically
+const result = await db.batch([listQuery, insertOp, updateOp]).execute();
+
+if (result.data) {
+  const [contactsList, insertResult, updateResult] = result.data;
+
+  console.log("Fetched contacts:", contactsList);
+  console.log("Inserted contact:", insertResult);
+  console.log("Updated user:", updateResult);
+}
+```
+
+### Transactional Behavior
+
+Batch operations are transactional for write operations (inserts, updates, deletes). If any operation in the batch fails, all write operations are rolled back:
+
+```typescript
+const result = await db
+  .batch([
+    db.from("users").insert({ username: "alice", email: "alice@example.com" }),
+    db.from("users").insert({ username: "bob", email: "bob@example.com" }),
+    db.from("users").insert({ username: "charlie", email: "invalid" }), // This fails
+  ])
+  .execute();
+
+if (result.error) {
+  // All three inserts are rolled back - no users were created
+  console.error("Batch failed:", result.error);
+}
+```
+
+**Note:** Batch operations automatically group write operations (POST, PATCH, DELETE) into changesets for transactional behavior, while read operations (GET) are executed individually within the batch.
+
+## Schema Management
+
+The library provides methods for managing database schema through the `db.schema` property. You can create and delete tables, add and remove fields, and manage indexes.
+
+### Creating Tables
+
+Create a new table with field definitions:
+
+```typescript
+import type { Field } from "@proofkit/fmodata";
+
+const fields: Field[] = [
+  {
+    name: "id",
+    type: "string",
+    primary: true,
+    maxLength: 36,
+  },
+  {
+    name: "username",
+    type: "string",
+    nullable: false,
+    unique: true,
+    maxLength: 50,
+  },
+  {
+    name: "email",
+    type: "string",
+    nullable: false,
+    maxLength: 255,
+  },
+  {
+    name: "age",
+    type: "numeric",
+    nullable: true,
+  },
+  {
+    name: "created_at",
+    type: "timestamp",
+    default: "CURRENT_TIMESTAMP",
+  },
+];
+
+const tableDefinition = await db.schema.createTable("users", fields);
+console.log(tableDefinition.tableName); // "users"
+console.log(tableDefinition.fields); // Array of field definitions
+```
+
+### Field Types
+
+The library supports various field types:
+
+**String Fields:**
+
+```typescript
+{
+  name: "username",
+  type: "string",
+  maxLength: 100, // Optional: varchar(100)
+  nullable: true,
+  unique: true,
+  default: "USER" | "USERNAME" | "CURRENT_USER", // Optional
+  repetitions: 5, // Optional: for repeating fields
+}
+```
+
+**Numeric Fields:**
+
+```typescript
+{
+  name: "age",
+  type: "numeric",
+  nullable: true,
+  primary: false,
+  unique: false,
+}
+```
+
+**Date Fields:**
+
+```typescript
+{
+  name: "birth_date",
+  type: "date",
+  default: "CURRENT_DATE" | "CURDATE", // Optional
+  nullable: true,
+}
+```
+
+**Time Fields:**
+
+```typescript
+{
+  name: "start_time",
+  type: "time",
+  default: "CURRENT_TIME" | "CURTIME", // Optional
+  nullable: true,
+}
+```
+
+**Timestamp Fields:**
+
+```typescript
+{
+  name: "created_at",
+  type: "timestamp",
+  default: "CURRENT_TIMESTAMP" | "CURTIMESTAMP", // Optional
+  nullable: false,
+}
+```
+
+**Container Fields:**
+
+```typescript
+{
+  name: "avatar",
+  type: "container",
+  externalSecurePath: "/secure/path", // Optional
+  nullable: true,
+}
+```
+
+### Adding Fields to Existing Tables
+
+Add new fields to an existing table:
+
+```typescript
+const newFields: Field[] = [
+  {
+    name: "phone",
+    type: "string",
+    nullable: true,
+    maxLength: 20,
+  },
+  {
+    name: "bio",
+    type: "string",
+    nullable: true,
+    maxLength: 1000,
+  },
+];
+
+const updatedTable = await db.schema.addFields("users", newFields);
+```
+
+### Deleting Tables and Fields
+
+Delete an entire table:
+
+```typescript
+await db.schema.deleteTable("old_table");
+```
+
+Delete a specific field from a table:
+
+```typescript
+await db.schema.deleteField("users", "old_field");
+```
+
+### Managing Indexes
+
+Create an index on a field:
+
+```typescript
+const index = await db.schema.createIndex("users", "email");
+console.log(index.indexName); // "email"
+```
+
+Delete an index:
+
+```typescript
+await db.schema.deleteIndex("users", "email");
+```
+
+### Complete Example
+
+Here's a complete example of creating a table with various field types:
+
+```typescript
+const fields: Field[] = [
+  // Primary key
+  {
+    name: "id",
+    type: "string",
+    primary: true,
+    maxLength: 36,
+  },
+
+  // String fields
+  {
+    name: "username",
+    type: "string",
+    nullable: false,
+    unique: true,
+    maxLength: 50,
+  },
+  {
+    name: "email",
+    type: "string",
+    nullable: false,
+    maxLength: 255,
+  },
+
+  // Numeric field
+  {
+    name: "age",
+    type: "numeric",
+    nullable: true,
+  },
+
+  // Date/time fields
+  {
+    name: "birth_date",
+    type: "date",
+    nullable: true,
+  },
+  {
+    name: "created_at",
+    type: "timestamp",
+    default: "CURRENT_TIMESTAMP",
+    nullable: false,
+  },
+
+  // Container field
+  {
+    name: "avatar",
+    type: "container",
+    nullable: true,
+  },
+
+  // Repeating field
+  {
+    name: "tags",
+    type: "string",
+    repetitions: 5,
+    maxLength: 50,
+  },
+];
+
+// Create the table
+const table = await db.schema.createTable("users", fields);
+
+// Later, add more fields
+await db.schema.addFields("users", [
+  {
+    name: "phone",
+    type: "string",
+    nullable: true,
+  },
+]);
+
+// Create an index on email
+await db.schema.createIndex("users", "email");
+```
+
+**Note:** Schema management operations require appropriate access privileges on your FileMaker account. Operations will throw errors if you don't have the necessary permissions.
+
 ## Advanced Features
 
 ### Type Safety
@@ -839,43 +1173,97 @@ db.from("users")
   .filter({ invalid: { eq: "john" } }); // TS Error
 ```
 
-### Required Fields
+### Required and Read-Only Fields
 
-Control which fields are required for insert and update operations:
+The library automatically infers which fields are required based on whether their validator allows `null` or `undefined`:
 
 ```typescript
 const usersBase = new BaseTable({
   schema: {
-    id: z.string(),
-    username: z.string(),
-    email: z.string(),
-    status: z.string(),
-    updatedAt: z.string().optional(),
+    id: z.string(), // Auto-required, auto-readOnly (idField)
+    username: z.string(), // Auto-required (not nullable)
+    email: z.string(), // Auto-required (not nullable)
+    status: z.string().nullable(), // Optional (nullable)
+    createdAt: z.string(), // Read-only system field
+    updatedAt: z.string().nullable(), // Optional
   },
-  idField: "id",
-  insertRequired: ["username", "email"], // Required on insert
-  updateRequired: ["status"], // Required on update
+  idField: "id", // Automatically excluded from insert/update
+  required: ["status"], // Make status required despite being nullable
+  readOnly: ["createdAt"], // Exclude createdAt from insert/update
 });
 
-// Insert requires username and email
+// Insert: username, email, and status are required
+// Insert: id and createdAt are excluded (cannot be provided)
 db.from("users").insert({
   username: "john",
   email: "john@example.com",
-  // updatedAt is optional
+  status: "active", // Required due to 'required' array
+  updatedAt: new Date().toISOString(), // Optional
 });
 
-// Update requires status
+// Update: all fields are optional except id and createdAt are excluded
 db.from("users")
   .update({
-    status: "active",
-    // other fields are optional
+    status: "active", // Optional
+    // id and createdAt cannot be modified
   })
   .byId("user-123");
 ```
 
+**Key Features:**
+
+- **Auto-inference:** Non-nullable fields are automatically required for insert
+- **Additional requirements:** Use `required` to make nullable fields required for new records
+- **Read-only fields:** Use `readOnly` to exclude fields from insert/update (e.g., timestamps)
+- **Automatic ID exclusion:** The `idField` is always read-only without needing to specify it
+- **Update flexibility:** All fields are optional for updates (except read-only fields)
+
+### Prefer: fmodata.entity-ids
+
+This library supports using FileMaker's internal field identifiers (FMFID) and table occurrence identifiers (FMTID) instead of names. This protects your integration from both field and table occurrence name changes.
+
+To enable this feature, simply define your schema with the `BaseTableWithIds` and `TableOccurrenceWithIds` classes. Behind the scenes, the library will transform your request and the response back to the names you specify in these schemas. This is an all-or-nothing feature. For it to work properly, you must define all table occurrences passed to a `Database` with the `TableOccurrenceWithIds` class.
+
+_Note for OttoFMS proxy: This feature requires version 4.14 or later of OttoFMS_
+
+How do I find these ids? They can be found in the XML version of the `$metadata` endpoint for your database, or you can calculate them using these [custom functions](https://github.com/rwu2359/CFforID) from John Renfrew
+
+#### Basic Usage
+
+```typescript
+import { BaseTableWithIds, TableOccurrenceWithIds } from "@proofkit/fmodata";
+import { z } from "zod/v4";
+
+// Define a base table with FileMaker field IDs
+const usersBase = new BaseTableWithIds({
+  schema: {
+    id: z.string(),
+    username: z.string(),
+    email: z.string().nullable(),
+    createdAt: z.string(),
+  },
+  idField: "id",
+  fmfIds: {
+    id: "FMFID:12039485",
+    username: "FMFID:34323433",
+    email: "FMFID:12232424",
+    createdAt: "FMFID:43234355",
+  },
+});
+
+// Create a table occurrence with a FileMaker table occurrence ID
+const usersTO = new TableOccurrenceWithIds({
+  name: "users",
+  baseTable: usersBase, // Must be a BaseTableWithIds
+  fmtId: "FMTID:12432533",
+});
+```
+
 ### Error Handling
 
-All operations return a `Result` type with either `data` or `error`:
+All operations return a `Result` type with either `data` or `error`. The library provides rich error types that help you handle different error scenarios appropriately.
+
+#### Basic Error Checking
 
 ```typescript
 const result = await db.from("users").list().execute();
@@ -890,6 +1278,277 @@ if (result.data) {
 }
 ```
 
+#### HTTP Errors
+
+Handle HTTP status codes (4xx, 5xx) with the `HTTPError` class:
+
+```typescript
+import { HTTPError, isHTTPError } from "@proofkit/fmodata";
+
+const result = await db.from("users").list().execute();
+
+if (result.error) {
+  if (isHTTPError(result.error)) {
+    // TypeScript knows this is HTTPError
+    console.log("HTTP Status:", result.error.status);
+
+    if (result.error.isNotFound()) {
+      console.log("Resource not found");
+    } else if (result.error.isUnauthorized()) {
+      console.log("Authentication required");
+    } else if (result.error.is5xx()) {
+      console.log("Server error - try again later");
+    } else if (result.error.is4xx()) {
+      console.log("Client error:", result.error.statusText);
+    }
+
+    // Access the response body if available
+    if (result.error.response) {
+      console.log("Error details:", result.error.response);
+    }
+  }
+}
+```
+
+#### Network Errors
+
+Handle network-level errors (timeouts, connection issues, etc.):
+
+```typescript
+import {
+  TimeoutError,
+  NetworkError,
+  RetryLimitError,
+  CircuitOpenError,
+} from "@proofkit/fmodata";
+
+const result = await db.from("users").list().execute();
+
+if (result.error) {
+  if (result.error instanceof TimeoutError) {
+    console.log("Request timed out");
+    // Show user-friendly timeout message
+  } else if (result.error instanceof NetworkError) {
+    console.log("Network connectivity issue");
+    // Show offline message
+  } else if (result.error instanceof RetryLimitError) {
+    console.log("Request failed after retries");
+    // Log the underlying error: result.error.cause
+  } else if (result.error instanceof CircuitOpenError) {
+    console.log("Service is currently unavailable");
+    // Show maintenance message
+  }
+}
+```
+
+#### Validation Errors
+
+When schema validation fails, you get a `ValidationError` with rich context:
+
+```typescript
+import { ValidationError, isValidationError } from "@proofkit/fmodata";
+
+const result = await db.from("users").list().execute();
+
+if (result.error) {
+  if (isValidationError(result.error)) {
+    // Access validation issues (Standard Schema format)
+    console.log("Validation failed for field:", result.error.field);
+    console.log("Issues:", result.error.issues);
+    console.log("Failed value:", result.error.value);
+  }
+}
+```
+
+**Validator-Agnostic Error Handling**
+
+The library uses [Standard Schema](https://github.com/standard-schema/standard-schema) to support any validation library (Zod, Valibot, ArkType, etc.). Following the same pattern as [uploadthing](https://github.com/pingdotgg/uploadthing), the `ValidationError.cause` property contains the normalized Standard Schema issues array:
+
+```typescript
+import { ValidationError } from "@proofkit/fmodata";
+
+const result = await db.from("users").list().execute();
+
+if (result.error instanceof ValidationError) {
+  // The cause property (ES2022 Error.cause) contains the Standard Schema issues array
+  // This is validator-agnostic and works with Zod, Valibot, ArkType, etc.
+  console.log("Validation issues:", result.error.cause);
+  console.log("Issues are also available directly:", result.error.issues);
+
+  // Both point to the same array
+  console.log(result.error.cause === result.error.issues); // true
+
+  // Access additional context
+  console.log("Failed field:", result.error.field);
+  console.log("Failed value:", result.error.value);
+
+  // Standard Schema issues have a normalized format
+  result.error.issues.forEach((issue) => {
+    console.log("Path:", issue.path);
+    console.log("Message:", issue.message);
+  });
+}
+```
+
+**Why Standard Schema Issues Instead of Original Validator Errors?**
+
+By using Standard Schema's normalized issue format in the `cause` property, the library remains truly validator-agnostic. All validation libraries that implement Standard Schema (Zod, Valibot, ArkType, etc.) produce the same issue structure, making error handling consistent regardless of which validator you choose.
+
+If you need validator-specific error formatting, you can still access your validator's methods during validation before the data reaches fmodata:
+
+```typescript
+import { z } from "zod";
+
+const userSchema = z.object({
+  email: z.string().email(),
+  age: z.number().min(0).max(150),
+});
+
+// Validate early if you need Zod-specific error handling
+const parseResult = userSchema.safeParse(userData);
+if (!parseResult.success) {
+  // Use Zod's error formatting
+  const formatted = parseResult.error.flatten();
+  console.log("Zod-specific formatting:", formatted);
+}
+```
+
+#### OData Errors
+
+Handle OData-specific protocol errors:
+
+```typescript
+import { ODataError, isODataError } from "@proofkit/fmodata";
+
+const result = await db.from("users").list().execute();
+
+if (result.error) {
+  if (isODataError(result.error)) {
+    console.log("OData Error Code:", result.error.code);
+    console.log("OData Error Message:", result.error.message);
+    console.log("OData Error Details:", result.error.details);
+  }
+}
+```
+
+#### Error Handling Patterns
+
+**Pattern 1: Using instanceof (like ffetch):**
+
+```typescript
+import {
+  HTTPError,
+  ValidationError,
+  TimeoutError,
+  NetworkError,
+} from "@proofkit/fmodata";
+
+const result = await db.from("users").list().execute();
+
+if (result.error) {
+  if (result.error instanceof TimeoutError) {
+    showTimeoutMessage();
+  } else if (result.error instanceof HTTPError) {
+    if (result.error.isNotFound()) {
+      showNotFoundMessage();
+    } else if (result.error.is5xx()) {
+      showServerErrorMessage();
+    }
+  } else if (result.error instanceof ValidationError) {
+    showValidationError(result.error.field, result.error.issues);
+  } else if (result.error instanceof NetworkError) {
+    showOfflineMessage();
+  }
+}
+```
+
+**Pattern 2: Using kind property (for exhaustive matching):**
+
+```typescript
+const result = await db.from("users").list().execute();
+
+if (result.error) {
+  switch (result.error.kind) {
+    case "TimeoutError":
+      showTimeoutMessage();
+      break;
+    case "HTTPError":
+      handleHTTPError(result.error.status);
+      break;
+    case "ValidationError":
+      showValidationError(result.error.field, result.error.issues);
+      break;
+    case "NetworkError":
+      showOfflineMessage();
+      break;
+    case "ODataError":
+      handleODataError(result.error.code);
+      break;
+    // TypeScript ensures exhaustive matching!
+  }
+}
+```
+
+**Pattern 3: Using type guards:**
+
+```typescript
+import {
+  isHTTPError,
+  isValidationError,
+  isODataError,
+  isNetworkError,
+} from "@proofkit/fmodata";
+
+const result = await db.from("users").list().execute();
+
+if (result.error) {
+  if (isHTTPError(result.error)) {
+    // TypeScript knows this is HTTPError
+    console.log("Status:", result.error.status);
+  } else if (isValidationError(result.error)) {
+    // TypeScript knows this is ValidationError
+    console.log("Field:", result.error.field);
+    console.log("Issues:", result.error.issues);
+  } else if (isODataError(result.error)) {
+    // TypeScript knows this is ODataError
+    console.log("Code:", result.error.code);
+  } else if (isNetworkError(result.error)) {
+    // TypeScript knows this is NetworkError
+    console.log("Network issue:", result.error.cause);
+  }
+}
+```
+
+#### Error Properties
+
+All errors include helpful metadata:
+
+```typescript
+if (result.error) {
+  // All errors have a timestamp
+  console.log("Error occurred at:", result.error.timestamp);
+
+  // All errors have a kind property for discriminated unions
+  console.log("Error kind:", result.error.kind);
+
+  // All errors have a message
+  console.log("Error message:", result.error.message);
+}
+```
+
+#### Available Error Types
+
+- **`HTTPError`** - HTTP status errors (4xx, 5xx) with helper methods (`is4xx()`, `is5xx()`, `isNotFound()`, etc.)
+- **`ODataError`** - OData protocol errors with code and details
+- **`ValidationError`** - Schema validation failures with issues, schema reference, and failed value
+- **`ResponseStructureError`** - Malformed API responses
+- **`RecordCountMismatchError`** - When `single()` or `maybeSingle()` expectations aren't met
+- **`TimeoutError`** - Request timeout (from ffetch)
+- **`NetworkError`** - Network connectivity issues (from ffetch)
+- **`RetryLimitError`** - Request failed after retries (from ffetch)
+- **`CircuitOpenError`** - Circuit breaker is open (from ffetch)
+- **`AbortError`** - Request was aborted (from ffetch)
+
 ### OData Annotations and Validation
 
 By default, the library automatically strips OData annotations fields (`@id` and `@editLink`) from responses. If you need these fields, you can include them by passing `includeODataAnnotations: true`: