@chaprola/mcp-server 1.6.4 → 1.8.0

package/references/ref-pivot.md

@@ -44,3 +44,14 @@ For COUNT: `"value": "department", "aggregate": "count"`
 SQL equivalent: `SELECT department, year, SUM(revenue) FROM sales GROUP BY department, year`
 
 Row and column totals are included automatically in the response.
+
+## TABULATE in Programs
+
+The `/query` pivot feature is also available in the Chaprola language via the TABULATE command:
+
+```chaprola
+TABULATE SALES SUM revenue FOR department VS year WHERE year GE "2020" INTO trends
+PRINT TABULATE trends AS CSV
+```
+
+TABULATE produces a matrix in memory — same cross-tabulation as `/query` pivot, but executed inside a program with dynamic PARAM values and chaining with QUERY results.

package/references/ref-programs.md

@@ -1,8 +1,16 @@
 # Chaprola Programs (.CS Source)
 
 ## Compile & Run
+
+**Best practice:** Start every program with `OPEN PRIMARY "filename" 0`. The compiler reads the format from the OPEN PRIMARY statement — no `primary_format` parameter needed. This makes programs self-documenting and eliminates compile-time guessing.
+
 ```bash
-POST /compile {userid, project, name: "REPORT", source: "...", primary_format: "STAFF", secondary_format?: "DEPTS"}
+# Preferred: source declares its own primary file
+POST /compile {userid, project, name: "REPORT", source: "OPEN PRIMARY \"STAFF\" 0\n..."}
+
+# Legacy: pass primary_format explicitly (still works, but OPEN PRIMARY is better)
+POST /compile {userid, project, name: "REPORT", source: "...", primary_format: "STAFF"}
+
 POST /run {userid, project, name: "REPORT", primary_file: "STAFF", record: 1, async?: true, nophi?: true}
 POST /run/status {userid, project, job_id}  # poll async jobs
 POST /publish {userid, project, name, primary_file, acl?: "public|authenticated|owner|token"}
@@ -12,33 +20,53 @@ POST /publish {userid, project, name, primary_file, acl?: "public|authenticated|
 
 | Prefix | Description |
 |--------|-------------|
-| `P` | Primary data file (current record) |
-| `S` | Secondary data file (current record) |
-| `U` | User buffer (output scratch) |
-| `X` | System text (date, time, filenames) |
+| `P` | Primary data file — access fields by name: `P.salary`, `P.last_name` |
+| `S` | Secondary data file — access fields by name: `S.dept`, `S.emp_id` |
+| `U` | User buffer (scratch for intermediate data) |
+| `X` | System text — access by property name: `X.username`, `X.utc_time`, `X.record_num` |
+
+### System Text Properties (X.)
+
+| Property | Description |
+|----------|-------------|
+| `X.year` | Year (four digits) |
+| `X.julian` | Julian date (1–366) |
+| `X.hour` | Hour (military time, 0–23) |
+| `X.minute` | Minute (0–59) |
+| `X.username` | Authenticated user (first 10 chars) |
+| `X.record_num` | Record number of primary file |
+| `X.display_file` | Display filename |
+| `X.data_file` | Data filename |
+| `X.proc_file` | Procedure filename |
+| `X.utc_time` | UTC datetime (ISO 8601, 20 chars) |
+| `X.elapsed` | Elapsed execution time (SSSSS.CC, 9 chars) |
+| `X.primary_modified` | Primary file Last-Modified (ISO 8601, 20 chars) |
+| `X.secondary_modified` | Secondary file Last-Modified (ISO 8601, 20 chars) |
 
 ## Language Essentials
 
 ```chaprola
-// Loop through records
+// Loop through records and print with concatenation
 DEFINE VARIABLE rec R41
 LET rec = 1
 100 SEEK rec
     IF EOF GOTO 900
-    MOVE P.name U.1 20          // copy field to output buffer
-    GET sal FROM P.salary        // numeric field → R variable
-    PUT sal INTO U.22 10 D 2     // R variable → formatted output
-    PRINT 0                      // output full U buffer, clear it
+    GET sal FROM P.salary
+    PRINT P.name + " — " + P.department + " — $" + sal
     LET rec = rec + 1
     GOTO 100
 900 END
 ```
 
-- `PRINT 0` — output entire U buffer and clear. `PRINT N` — output exactly N chars.
-- `MOVE BLANKS U.1 80` — clear a region. `MOVE "literal" U.1 7` — move literal.
-- `IF EQUAL "text" U.50 4 GOTO 200` — compare literal to memory location.
-- `U.name` — named positions (auto-allocated by compiler): `MOVE P.name U.name 20`
-- `DEFINE VARIABLE counter R41` — alias R-variable. **Use R41-R50** (R1-R40 reserved for HULDRA).
+- **PRINT concatenation (preferred):** `PRINT P.name + " earns " + R41` — auto-trims fields, auto-formats numbers, auto-flushes.
+- `PRINT P.fieldname` — output a single field (auto-flush, auto-trim).
+- `PRINT "literal"` — output a literal string (auto-flush).
+- `PRINT 0` — output entire U buffer (less common, for columnar reports).
+- `CLEAR U` — clear entire user buffer. `CLEAR P` / `CLEAR S` — clear primary/secondary region.
+- `MOVE BLANKS P.notes` — clear a single field (length auto-filled).
+- `MOVE "Active" P.status` — write literal to field (auto-padded to field width).
+- `IF EQUAL "text" P.status GOTO 200` — compare literal to field.
+- `DEFINE VARIABLE counter R41` — alias R-variable. **Use R41-R99** (R1-R40 reserved for HULDRA).
 
 ## PUT Format Codes
 
@@ -49,7 +77,7 @@ LET rec = 1
 | `I` | Integer (right-justified) | `  1234` |
 | `E` | Scientific notation | `1.23E+03` |
 
-Syntax: `PUT R41 INTO U.30 10 D 2` — (R-var, location, width, format, decimals)
+Syntax: `PUT R41 INTO P.salary D 2` — (R-var, location, width auto-filled from field name, format, decimals)
 
 ## Math
 
@@ -59,14 +87,24 @@ LET R43 = EXP R41       // also: LOG, SQRT, ABS
 LET R44 = POW R41 R42   // R41^R42
 ```
 
+## Date Arithmetic
+
+```chaprola
+GET DATE R41 FROM X.primary_modified   // parse timestamp → epoch seconds
+GET DATE R42 FROM X.utc_time           // current UTC time
+LET R43 = R42 - R41                    // difference in seconds
+LET R43 = R43 / 86400                  // convert to days
+PUT DATE R42 INTO U.1 20               // write epoch as ISO 8601 string
+```
+
 ## Secondary Files (FIND/JOIN)
 
 ```chaprola
 OPEN "DEPARTMENTS" 0              // open secondary file
-FIND match FROM S.dept_code 3 USING P.dept_code
+FIND match FROM S.dept_code USING P.dept_code
 IF match EQ 0 GOTO 200           // 0 = no match
 READ match                        // load matched record
-MOVE S.dept_name U.30 15
+PRINT P.name + " — " + S.dept_name
 WRITE match                       // write back if modified
 CLOSE                             // flush + close
 ```
@@ -81,5 +119,74 @@ LET lvl = PARAM.level            // numeric param → R variable
 Publish, then call: `POST /report?userid=X&project=Y&name=Z&deck=kanji&level=3`
 Discover params: `POST /report/params {userid, project, name}`
 
+## QUERY Command
+
+QUERY filters, selects, and reorganizes data inside a Chaprola program — the same power as the `/query` API endpoint, but as a language command.
+
+**Output is a .QR file (read-only snapshot).** Cannot be modified with INSERT, UPDATE, or DELETE. Use the original .DA file for writes. R20 is set to the number of matched records.
+
+```chaprola
+// Filter + column select
+QUERY STAFF FIELDS name, salary INTO HIGH_PAID WHERE salary GT 80000
+
+// Dynamic WHERE with params and R-variables
+QUERY flashcards INTO results WHERE level EQ PARAM.level
+QUERY data INTO subset WHERE score GE R5 AND category EQ PARAM.type
+
+// BETWEEN with dynamic bounds
+QUERY data INTO results WHERE age BETWEEN PARAM.min_age PARAM.max_age
+
+// Cross-file filtering (IN/NOT IN) — one per QUERY
+QUERY flashcards INTO new_cards WHERE kanji NOT IN progress.kanji
+
+// GROUP BY
+QUERY orders INTO summary WHERE year EQ "2026" GROUP BY region COUNT, SUM total ORDER BY SUM_TOTAL DESC LIMIT 5
+
+// FROM syntax (alternative to INTO)
+QUERY results FROM STAFF FIELDS name, salary WHERE dept EQ PARAM.dept
+
+// OPEN with WHERE (filter directly into file handle)
+OPEN SECONDARY customers WHERE customer_id IN orders.customer_id
+```
+
+### QUERY Errors
+- **Missing source file:** FOERR flag set, QUERY skipped. Program can check FOERR and branch. (R20 retains its prior value.)
+- **Missing IN-file:** NOT IN treats as empty set (all records pass). IN treats as empty set (no records pass). This is intentional — a new user with no progress file gets all flashcards.
+- **Missing PARAM:** Silently replaced with blank (string) or 0.0 (numeric). Not a hard error — program continues. Check param warnings in the response for diagnostics.
+- **Zero matches:** Not an error. R20 = 0, output .QR is empty.
+
+### QUERY Limits
+- One index lookup per QUERY (first EQ condition only)
+- One IN/NOT IN per QUERY
+- No nested QUERY — QUERY is a statement, not an expression
+- Output is always a new file — QUERY never modifies the source
+- FIELDS and GROUP BY are mutually exclusive
+
+## TABULATE Command
+
+TABULATE produces cross-tabulation matrices inside a program — the language equivalent of `/query` pivot. Result is in-memory only (not written to S3).
+
+```chaprola
+TABULATE sales SUM revenue FOR region VS quarter WHERE year EQ "2026" INTO matrix
+PRINT TABULATE matrix AS CSV     // CSV output for charting
+PRINT TABULATE matrix AS JSON    // JSON matrix for web apps
+PRINT TABULATE matrix AS TABLE   // text table for preview
+```
+
+Aggregates: COUNT, SUM, AVG, MIN, MAX. Multiple aggregates: `TABULATE data COUNT, SUM total FOR row VS col ...`
+
+## File Properties
+
+```chaprola
+LET R1 = orders.RECORDCOUNT     // record count of any loaded file
+IF R1 EQ 0 GOTO no_data
+```
+
+## INDEX Command
+
+```chaprola
+INDEX STAFF ON department        // creates STAFF.DEPARTMENT.IDX
+```
+
 ## Common Field Widths
 ISO datetime: 20, UUID: 36, email: 50, short ID: 8-12, dollar: 10, phone: 15.

package/references/ref-query.md

@@ -38,3 +38,54 @@ Types: `inner`, `left`, `right`, `full`. Optional `pre_sorted: true` for merge j
 - `POST /update-record {userid, project, file, where: [...], set: {field: "value"}}`
 - `POST /delete-record {userid, project, file, where: [...]}`
 - `POST /consolidate {userid, project, file}` — merge .MRG into .DA
+
+## QUERY in Programs
+
+The QUERY language command does the same thing as the `/query` API but inside a Chaprola program. Use it to filter, select, and reorder data without leaving the runtime.
+
+```chaprola
+// In a program, QUERY replaces /query API calls
+QUERY STAFF FIELDS name, salary INTO TOP_EARNERS WHERE salary GT 80000 ORDER BY salary DESC LIMIT 10
+```
+
+The result is a `.QR` file (read-only snapshot) that can be opened as a secondary file or used in subsequent QUERY commands. R20 is set to the number of matched records. INSERT, UPDATE, and DELETE operations are rejected on .QR files.
+
+If the source file doesn't exist, the FOERR flag is set and the QUERY is skipped. If an IN/NOT IN reference file doesn't exist, it's treated as an empty set (NOT IN = all pass, IN = none pass).
+
+## Clustered Sort Columns
+
+Import with `sort_columns` to create self-indexing files. The data is physically sorted by the key columns at import time, enabling binary search on the clustered key without a separate .IDX file.
+
+```json
+POST /import {
+  "userid": "...", "project": "...", "name": "STAFF",
+  "sort_columns": ["department", "name"],
+  "data": [...]
+}
+```
+
+- The .F file marks KEY fields (`KEY:1`, `KEY:2`, etc.)
+- QUERY automatically uses binary search on clustered keys
+- No separate .IDX needed for primary access patterns
+
+## split_by on /import
+
+Split a dataset into per-group data files at import time. One `.DA` file is created per distinct value of the split field, sharing a single `.F` format file.
+
+```json
+POST /import {
+  "userid": "...", "project": "...", "name": "orders",
+  "split_by": "region",
+  "data": [...]
+}
+```
+
+Produces files like `orders/east.DA`, `orders/west.DA`, etc. Access with dynamic filenames in a program:
+
+```chaprola
+OPEN PRIMARY orders/PARAM.region
+```
+
+## BAA and Site Keys
+
+The `/query` API endpoint requires BAA signing. Site keys inherit this requirement. For public-facing web apps, use published reports (`/report`) instead of `/query` for all read operations. Move filtering and aggregation logic into Chaprola programs using the QUERY and TABULATE language commands, publish them, and call `/report` from the frontend. Reserve site keys for write-only operations like `/insert-record`.