langfx.js 0.1.0-alpha.0

package/docs/PROMPT_PARITY.md

@@ -0,0 +1,379 @@
+# LfQuery prompt parity report
+
+Python source: `0a17208b4290e470e398892c6232819225289cc0` (clean checkout), protocol `json:1.0`.
+
+Python prompts were executed, not transcribed. Both runtimes were also checked against their query wrapper. Inputs are equivalent already-rendered messages, with equivalent integer/object schemas. No inference or credentials were used.
+
+Normalization only trims section boundaries and converts CRLF to LF. It does not erase roles, labels, examples, or schema representations.
+
+| Case | Exact parity | Normalized parity | Protocol-adapted parity | Python roles | JS roles |
+| --- | --- | --- | --- | --- | --- |
+| integer | FAIL | FAIL | PASS | system, user | system, user |
+| object | FAIL | FAIL | PASS | system, user | system, user |
+| system-message | FAIL | FAIL | PASS | system, user | system, user |
+| few-shot | FAIL | FAIL | PASS | system, user | system, user |
+| few-shot-and-system | FAIL | FAIL | PASS | system, user | system, user |
+
+## Differences
+
+- Prompt structure is aligned: protocol preamble and demonstration, examples, request schema, caller instructions, then labeled user request and output cue.
+- Fixed adaptations are listed in `tests/fixtures/prompt-protocol-adaptations.json`. They replace only the protocol-specific preamble, known schema representations, and example result envelopes; roles, order, and user text remain intact.
+- Deliberate protocol differences: Python’s JSON protocol uses a `result` envelope, Python-style schema descriptions, and `_type` instructions. JS uses standard JSON Schema and direct output values. Literal prompt equality requires resolving these differences, not merely copying the preamble.
+
+This comparison does not cover media, custom class templates, native output, Python-protocol prompts, provider wire encoding, or response quality. Exact equality is still not claimed; it does not claim full semantic equivalence.
+
+## Captured prompts
+
+### integer
+
+**Python**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. If the schema has `_type`, carry it over to the JSON output.
+  2. If a field from the schema cannot be extracted from the response, use null as the JSON value.
+
+SCHEMA:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": int}}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": 2}}
+
+SCHEMA:
+  {"result": int}
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```
+
+**TypeScript**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. Return only JSON matching the JSON Schema, without Markdown fences or a result wrapper.
+  2. Use null only when the schema permits it; do not invent missing facts.
+
+SCHEMA:
+  {"type":"object","properties":{"final_answer":{"type":"integer"}},"required":["final_answer"]}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"final_answer":2}
+
+SCHEMA:
+  {"type":"integer"}
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```
+
+### object
+
+**Python**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. If the schema has `_type`, carry it over to the JSON output.
+  2. If a field from the schema cannot be extracted from the response, use null as the JSON value.
+
+SCHEMA:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": int}}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": 2}}
+
+SCHEMA:
+  {"result": {"answer": int}}
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```
+
+**TypeScript**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. Return only JSON matching the JSON Schema, without Markdown fences or a result wrapper.
+  2. Use null only when the schema permits it; do not invent missing facts.
+
+SCHEMA:
+  {"type":"object","properties":{"final_answer":{"type":"integer"}},"required":["final_answer"]}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"final_answer":2}
+
+SCHEMA:
+  {"type":"object","properties":{"answer":{"type":"integer"}},"required":["answer"]}
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```
+
+### system-message
+
+**Python**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. If the schema has `_type`, carry it over to the JSON output.
+  2. If a field from the schema cannot be extracted from the response, use null as the JSON value.
+
+SCHEMA:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": int}}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": 2}}
+
+SCHEMA:
+  {"result": int}
+
+Be concise.
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```
+
+**TypeScript**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. Return only JSON matching the JSON Schema, without Markdown fences or a result wrapper.
+  2. Use null only when the schema permits it; do not invent missing facts.
+
+SCHEMA:
+  {"type":"object","properties":{"final_answer":{"type":"integer"}},"required":["final_answer"]}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"final_answer":2}
+
+SCHEMA:
+  {"type":"integer"}
+
+Be concise.
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```
+
+### few-shot
+
+**Python**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. If the schema has `_type`, carry it over to the JSON output.
+  2. If a field from the schema cannot be extracted from the response, use null as the JSON value.
+
+SCHEMA:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": int}}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": 2}}
+
+SCHEMA:
+  {"result": int}
+
+INPUT_OBJECT:
+  1 + 1?
+
+JSON:
+  {"result": 2}
+
+SCHEMA:
+  {"result": int}
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```
+
+**TypeScript**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. Return only JSON matching the JSON Schema, without Markdown fences or a result wrapper.
+  2. Use null only when the schema permits it; do not invent missing facts.
+
+SCHEMA:
+  {"type":"object","properties":{"final_answer":{"type":"integer"}},"required":["final_answer"]}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"final_answer":2}
+
+SCHEMA:
+  {"type":"integer"}
+
+INPUT_OBJECT:
+  1 + 1?
+
+JSON:
+  2
+
+SCHEMA:
+  {"type":"integer"}
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```
+
+### few-shot-and-system
+
+**Python**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. If the schema has `_type`, carry it over to the JSON output.
+  2. If a field from the schema cannot be extracted from the response, use null as the JSON value.
+
+SCHEMA:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": int}}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"result": {"_type": "langfx.structured.query.Answer", "final_answer": 2}}
+
+SCHEMA:
+  {"result": int}
+
+INPUT_OBJECT:
+  1 + 1?
+
+JSON:
+  {"result": 2}
+
+SCHEMA:
+  {"result": int}
+
+Be concise.
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```
+
+**TypeScript**
+
+```text
+[system]
+Please respond to the last INPUT_OBJECT with JSON according to SCHEMA:
+
+INSTRUCTIONS:
+  1. Return only JSON matching the JSON Schema, without Markdown fences or a result wrapper.
+  2. Use null only when the schema permits it; do not invent missing facts.
+
+SCHEMA:
+  {"type":"object","properties":{"final_answer":{"type":"integer"}},"required":["final_answer"]}
+
+INPUT_OBJECT:
+  1 + 1 =
+
+JSON:
+  {"final_answer":2}
+
+SCHEMA:
+  {"type":"integer"}
+
+INPUT_OBJECT:
+  1 + 1?
+
+JSON:
+  2
+
+SCHEMA:
+  {"type":"integer"}
+
+Be concise.
+
+[user]
+INPUT_OBJECT:
+  What is 6 * 7?
+
+JSON:
+
+```

package/docs/PYTHON_PROTOCOL.md

@@ -0,0 +1,109 @@
+# Python-style constructor protocol
+
+Python-style output is the default for prompt-based structured `query`, `Session.query`, `Mapping`, and `LfQuery`. Set `protocol: 'json'` explicitly for JSON Schema/Zod schemas without Python codecs. No schema means ordinary text output, unchanged. JSON Schema also supports unions; the benefit here is named constructor syntax and explicit construction of TypeScript class instances.
+
+```ts
+import * as lf from 'langfx.js';
+
+class Foo { constructor(readonly text: string) {} }
+class Bar { constructor(readonly count: number) {} }
+
+const FooSchema = lf.python.classSchema({
+  name: 'Foo',
+  fields: { text: lf.python.str },
+  create: fields => new Foo(fields.text),
+  isInstance: (value): value is Foo => value instanceof Foo,
+  decompose: value => ({ text: value.text }),
+});
+const BarSchema = lf.python.classSchema({
+  name: 'Bar',
+  fields: { count: lf.python.int },
+  create: fields => new Bar(fields.count),
+  isInstance: (value): value is Bar => value instanceof Bar,
+  decompose: value => ({ count: value.count }),
+});
+const FooOrBar = lf.python.union(FooSchema, BarSchema);
+const result = await lf.query('Choose a response.', FooOrBar, {
+  lm, // protocol: 'python' is the default
+}); // Foo | Bar
+if (result instanceof Foo) console.log(result.text);
+```
+
+Schemas can also be assigned to a class's static `schema` property, allowing `lf.python.union(Foo, Bar)`. JavaScript cannot overload the `|` operator for runtime schemas.
+
+The prompt renders `Union[Foo, Bar]` and Python class declarations. Responses such as `Foo(text="hello")` or `output = Bar(count=7)` are parsed locally. One complete Markdown fence (optionally tagged `python`) is accepted. Nested constructors, `list(schema)`, string-keyed `dict(schema)`, `str`, `int`, `float`, `bool`, and `none` are supported. `union(schema, none)` expresses nullable values. Fields are required unless a validated default is supplied; extra fields reject unless an `extraFields` schema is supplied. Integer values must be safe JavaScript integers.
+
+The parser accepts only data literals and registered keyword constructors. It never uses eval/Function or runs generated Python/JavaScript. Imports, attributes, positional arguments, arithmetic, comprehensions, executable statements, and unknown constructors reject. Parsing is bounded to 1,048,576 UTF-16 units and depth 64. Class factories and schema codecs are application code and must be pure; union validation may probe alternatives. An entire response's syntax is checked before decoding invokes factories. Type validation of nested structures may invoke inner factories before an outer validation failure.
+
+`decompose` and `isInstance` support few-shot examples: provide actual instances in `MappingExample.output`. The formatter renders constructor syntax and the example is checked against its schema. `lf.python.parse`, compact `format`, expanded fenced `formatExample`, and `schemaText` expose the same local operations. These schemas also carry JSON Schema; Unions try alternatives in declaration order, including overlapping numeric alternatives. Constructor names distinguish otherwise identical class field shapes.
+
+This is a supported subset, not a Python interpreter or a promise of parity for all schemas. A missing-value object model, automatic Python enum/class introspection, arbitrary Zod-to-Python conversion, repair/code execution, and complete Python incremental-parser parity are not implemented. Native structured output is JSON-specific and cannot be combined with `protocol: 'python'`. Structured `queryStream` emits raw `textDelta` events and best-effort frozen field previews in Python mode, then parses and validates once at completion. See [streaming](STREAMING.md) for the supported preview syntax and its limits. `nativeStructuredOutput: true` selects JSON when protocol is omitted. Existing JSON prompt fixtures and their documented adaptations are unchanged.
+
+## Python comparison
+
+`npm run check:python-parity` checks nine real Python captures: integer, class, both union branches, nested classes, few-shot examples, a nullable class list, nested examples, and examples with a system message. Full role/text arrays match exactly (9/9), without normalization, as do parsed values/class identity. See [the full comparison](PYTHON_PROTOCOL_PARITY.md).
+
+Regenerate captures with `../langfx/.venv/bin/python scripts/capture-python-protocol.py ../langfx`. The fixture records the actual Python commit and dirty state. Normal checks require only Node. The captured Python checkout requires explicit PyGlove ValueSpecs for top-level union cases; raw `Foo | Bar` annotations are rejected in its prompt-schema conversion path.
+
+## Broader feature evaluation
+
+The nine exact-prompt cases do not imply general Python schema parity. See [the comprehensive feature evaluation](PYTHON_SCHEMA_EVALUATION.md) for constraints, defaults, string grammar, inheritance, recursive types, and observed validation differences. `npm run evaluate:schemas` reports known gaps rather than treating them as passing parity tests.
+
+## Schema features
+
+The built-in helpers support the following explicit definitions:
+
+```ts
+const p = lf.python;
+const Choice = p.literal('accept', 'reject');
+const Score = p.integer({ min: 0, max: 10 });
+const Confidence = p.number({ min: 0, max: 1 });
+const Label = p.string({ pattern: '[A-Z]+' });
+const Scores = p.list(Score, { minSize: 1, maxSize: 5 });
+const Pair = p.tuple(p.str, Score); // TypeScript tuple array
+const Result = p.record({ decision: Choice, score: Score });
+```
+
+`classSchema` accepts `defaults`, `description`, `fieldDescriptions`, `excludeFromPrompt`, and `extraFields`. Defaults are validated and copied through the field codec for each new instance. Excluded fields need defaults and are omitted from schema declarations; serialized values still contain them, matching Python. Use `extraFields: p.any` to allow additional data-valued keywords, or provide a narrower schema.
+
+Use `withSubclasses(BaseSchema, ChildSchema)` to explicitly allow child constructors. Declare inherited fields in each child schema. The query graph renders registered child declarations as `class Child(Base)`. Standalone child schemas retain their own declarations; registration does not discover classes globally. Conflicting parent registrations reject. `lazy('Node', () => NodeSchema)` provides named recursive references; define its target before rendering or parsing. Exported JSON Schema includes the reachable `$defs`.
+
+The data parser supports tuples, hexadecimal/octal/binary and underscore-separated numbers, raw/triple/adjacent strings, and common Unicode/control escapes. Integer schemas reject floating literals such as `7.0`. Booleans remain distinct from numbers, integers must be safe JavaScript integers, and duplicate dictionary keys reject. Regex constraints use JavaScript syntax with a start anchor; Python-specific regex syntax is not portable. `any` accepts data literals without executing constructors; JavaScript values do not preserve Python's numeric or tuple identity when a schema does not specify it.
+
+## Sets and unknown values
+
+```ts
+const Numbers = lf.python.set(lf.python.int);
+const numbers = lf.python.parse(Numbers, '{1, 2, 1}'); // Set<number>
+const empty = lf.python.parse(Numbers, 'set()');
+
+const MaybeCount = lf.python.allowUnknown(lf.python.int);
+const count = lf.python.parse(MaybeCount, 'UNKNOWN'); // number | typeof lf.python.UNKNOWN
+if (count === lf.python.UNKNOWN) {
+  // The model could not provide a value.
+}
+```
+
+`set()` accepts primitive elements; `set(itemSchema)` adds element validation. This is stricter than the evaluated Python checkout, which renders `set[int]` as `set` and does not enforce its element annotation. Collections, constructors, and UNKNOWN cannot be set elements in this implementation. Boolean/numeric duplicates follow Python equality (`True` equals `1`). Formatting follows JavaScript insertion order, so exact set text may differ from Python. JSON mode represents sets as arrays and parses them into `Set` instances.
+
+`any` accepts `UNKNOWN` directly, including nested occurrences. Typed schemas require `allowUnknown` at each field that may contain the marker, preserving accurate TypeScript return types. The wrapper retains the underlying prompt annotation and JSON Schema; UNKNOWN has no JSON wire representation. Unlike Python, ordinary `int`, class, and collection schemas do not silently accept UNKNOWN. Bare `MISSING` and `Missing()` reject in both response parsers; defaults remain the way to handle omitted fields.
+
+## Dataclass-style examples
+
+For an explicitly registered class matching a Python dataclass, supply its signature description and request compact examples:
+
+```ts
+class Data { constructor(readonly count: number) {} }
+const DataSchema = lf.python.classSchema({
+  name: 'Data',
+  fields: { count: lf.python.int },
+  description: 'Data(count: int)',
+  exampleFormat: 'compact',
+  create: fields => new Data(fields.count),
+  isInstance: (value): value is Data => value instanceof Data,
+  decompose: value => ({ count: value.count }),
+});
+// formatExample(DataSchema, new Data(1)) emits fenced `output = Data(count=1)`.
+```
+
+The setting affects few-shot/example formatting and propagates through collection schemas, unions, subclass registration, unknown-value wrappers, and recursive references. Other classes retain expanded formatting by default. Inside a compact constructor, nested values also render compactly. Conflicting example styles for the same class name reject. This is explicit presentation metadata; no Python dataclass introspection or automatic signature/default generation is performed.