eyelang 1.7.2 → 1.7.3

package/docs/guide.md

@@ -497,7 +497,7 @@ node test/run-regression.mjs
 node test/run-examples.mjs
 ```
 
-The conformance suite lives in [`test/conformance/`](../test/conformance/) as one flat eyelang corpus. Each case is a small `.eye` program with an exact expected stdout `.eye` file, and some cases also include a goal file for testing the embeddable solver, so other implementations can reuse the same cases. The suite covers the standard language surface from the language reference, including reusable built-ins. The regression suite lives in [`test/run-regression.mjs`](../test/run-regression.mjs) and covers CLI regressions, the public JavaScript API, and white-box invariants for parser, unification, and indexing behavior.
+The conformance suite lives in [`test/conformance/`](../test/conformance/) as a file-based eyelang corpus. Positive cases pair `cases/<name>.eye` with exact expected stdout under `expected/<name>.eye`; negative cases pair `errors/<name>.eye` with exact expected error text under `expected-errors/<name>.txt`; warning cases pair `warnings/<name>.eye` with exact `--warnings` stdout and stderr files under `expected-warnings/`. Cases may be grouped in category directories such as `atoms/`, `variables/`, `negation/`, and `syntax/`, so another implementation can reuse the same corpus as an executable language contract. The suite covers the standard language surface from the language reference, including reusable built-ins, standard errors, and standard warnings. The regression suite lives in [`test/run-regression.mjs`](../test/run-regression.mjs) and covers CLI regressions, the public JavaScript API, and white-box invariants for parser, unification, and indexing behavior.
 
 ## Development and release
 

package/docs/language-reference.md

@@ -702,7 +702,7 @@ A conforming eyelang implementation supports the standard language described abo
 
 Browser execution, package layout, CLI URL loading, and any implementation-specific built-ins described in host documentation are outside this conformance surface unless separately standardized.
 
-Conformance cases live in the repository under `test/conformance/`. They are run by `npm test` before the example suite, and can be run alone with `node test/run-conformance.mjs`. Each case has an input program under `conformance/cases/` and an exact expected standard-output file under `conformance/expected/`; both use `.eye` so expected output remains eyelang-readable.
+Conformance cases live in the repository under `test/conformance/`. They are run by `npm test` before the example suite, and can be run alone with `node test/run-conformance.mjs`. Positive cases have input programs under `conformance/cases/` and exact expected standard-output files under `conformance/expected/`; both use `.eye` so expected output remains eyelang-readable. Expected-error cases live under `conformance/errors/` with exact messages under `conformance/expected-errors/`. Expected-warning cases live under `conformance/warnings/` with exact `--warnings` stdout and stderr files under `conformance/expected-warnings/`.
 
 ## 15. Relationship to ISO Prolog
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "eyelang",
-  "version": "1.7.2",
+  "version": "1.7.3",
   "description": "A small Prolog-like logic programming language for rules, goals, answers, and proofs.",
   "type": "module",
   "main": "./index.js",

package/test/conformance/README.md

@@ -2,12 +2,25 @@
 
 This directory contains the executable conformance cases for the Eyelang language and reference engine. The normative language description is in the [Eyelang language reference](../../../docs/language-reference.md).
 
-The suite is intentionally file-based so another implementation can run the same programs and compare exact standard output. A case consists of:
+The suite is intentionally file-based so another implementation can run the same programs and compare exact standard output, expected errors, and expected warnings. The conformance corpus is part of the public language contract, not just an implementation smoke test.
+
+A normal positive case consists of:
 
 - `conformance/cases/<name>.eye` — input program;
 - `conformance/expected/<name>.eye` — exact expected standard output, stored as eyelang-readable facts.
 
-The current runner compares standard output from normal execution. Proof explanations are opt-in in the CLI and are not part of these conformance goldens. Standard error, performance, and resource limits are outside this suite.
+Expected-error cases consist of:
+
+- `conformance/errors/<name>.eye` — input program that must fail during parsing or execution;
+- `conformance/expected-errors/<name>.txt` — exact expected error message followed by a newline.
+
+Expected-warning cases consist of:
+
+- `conformance/warnings/<name>.eye` — input program run through the CLI with `--warnings`;
+- `conformance/expected-warnings/<name>.eye` — exact expected standard output;
+- `conformance/expected-warnings/<name>.txt` — exact expected standard error.
+
+Case names may be nested in category directories such as `atoms/`, `variables/`, `negation/`, or `syntax/`. Expected files mirror the same relative path.
 
 ## Running the suite
 
@@ -23,21 +36,23 @@ Run only the conformance suite:
 node test/run-conformance.mjs
 ```
 
-Run matching conformance cases by passing a filename fragment:
+Run matching conformance cases by passing a filename or directory fragment:
 
 ```sh
 node test/run-conformance.mjs reusable
 node test/run-conformance.mjs 092_scalar_string_conversions
+node test/run-conformance.mjs variables/
+node test/run-conformance.mjs error/variables
 ```
 
-The runner executes materialized programs in-process through the public JavaScript API so small conformance cases avoid measuring Node startup overhead.
+The runner executes normal materialized programs in-process through the public JavaScript API so small conformance cases avoid measuring Node startup overhead. Warning cases intentionally use the CLI because warning output is a host-interface contract.
 
 ## Scope
 
-The conformance corpus is a single eyelang suite. It covers the standard language described by the language reference: lexical syntax, facts, definite clauses, first-order terms, lists, conjunction, structured unification, left-to-right goal-directed proof search, materialized output, read-back printing, standard built-ins, declarations, and standard host behavior.
+The conformance corpus is a single eyelang suite. It covers the standard language described by the language reference: lexical syntax, facts, definite clauses, first-order terms, lists, conjunction, structured unification, left-to-right goal-directed proof search, materialized output, read-back printing, standard built-ins, declarations, warnings, errors, and standard host behavior.
 
 The suite deliberately does not separate `core` and `extension` profiles. Reusable built-ins such as arithmetic, strings, lists, aggregation, context terms, term inspection, and search control are part of the standard eyelang conformance surface. Implementation-specific built-ins may still exist in downstream hosts, but they should have their own tests outside this corpus unless they are standardized.
 
 ## Updating expected output
 
-There is no committed auto-accept mode. To update an expected file, run the matching case with the conformance runner, inspect the result, and replace the corresponding file under `conformance/expected/` deliberately.
+There is no committed auto-accept mode. To update an expected file, run the matching case with the conformance runner, inspect the result, and replace the corresponding file under `conformance/expected/`, `conformance/expected-errors/`, or `conformance/expected-warnings/` deliberately.

package/test/conformance/cases/atoms/graphic_angle_atom_stays_graphic.eye

@@ -0,0 +1,4 @@
+% Graphic atoms that are not absolute IRIs remain graphic atoms.
+materialize(answer, 1).
+operator(<=>).
+answer(?op) :- operator(?op).

package/test/conformance/cases/atoms/iri_atoms_in_lists.eye

@@ -0,0 +1,4 @@
+% IRI atoms can appear anywhere ordinary atoms can appear, including lists.
+materialize(answer, 1).
+route([<urn:example:a>, <urn:example:b>, <urn:example:c>]).
+answer(?route) :- route(?route).

package/test/conformance/cases/atoms/iri_quoted_absolute_readback.eye

@@ -0,0 +1,5 @@
+% Quoted absolute IRI atoms read back in angle-bracket form.
+materialize(answer, 1).
+item('https://example.org/alice').
+item('urn:example:bob').
+answer(?iri) :- item(?iri).

package/test/conformance/cases/atoms/iri_quoted_and_angle_unify.eye

@@ -0,0 +1,3 @@
+% Angle-bracket and quoted spellings denote the same absolute IRI atom.
+materialize(answer, 1).
+answer(ok) :- eq(<urn:example:a>, 'urn:example:a').

package/test/conformance/cases/atoms/zero_arity_atom_roundtrip.eye

@@ -0,0 +1,5 @@
+% Arity-zero data is represented as atoms, never zero-arity compounds.
+materialize(answer, 2).
+answer(name, ?name) :- compound_name_arguments(nil, ?name, ?args).
+answer(args, ?args) :- compound_name_arguments(nil, ?name, ?args).
+answer(term, ?term) :- compound_name_arguments(?term, nil, []).

package/test/conformance/cases/declarations/memoize_is_ordinary_fact.eye

@@ -0,0 +1,4 @@
+% `memoize/2` is no longer a declaration, but it remains an ordinary fact.
+materialize(answer, 2).
+memoize(path, 2).
+answer(?name, ?arity) :- memoize(?name, ?arity).

package/test/conformance/cases/declarations/mode_determinism_are_facts.eye

@@ -0,0 +1,8 @@
+% Advisory declarations are also ordinary facts that programs can inspect.
+mode(path, 2, [in, out]).
+semidet(edge, 2).
+det(root, 1).
+materialize(answer, 2).
+answer(mode, ?modes) :- mode(path, 2, ?modes).
+answer(semidet, edge) :- semidet(edge, 2).
+answer(det, root) :- det(root, 1).

package/test/conformance/cases/negation/stratified_negation_answers.eye

@@ -0,0 +1,6 @@
+% Stratified negation is portable and produces ordinary answers.
+materialize(open, 1).
+place(a).
+place(b).
+closed(b).
+open(?x) :- place(?x), not(closed(?x)).

package/test/conformance/cases/table/table_does_not_change_answers.eye

@@ -0,0 +1,7 @@
+% `table/2` is a search-control declaration and does not change answers.
+materialize(path, 2).
+table(path, 2).
+edge(a, b).
+edge(b, c).
+path(?x, ?y) :- edge(?x, ?y).
+path(?x, ?z) :- edge(?x, ?y), path(?y, ?z).

package/test/conformance/cases/variables/question_anonymous_not_reused.eye

@@ -0,0 +1,6 @@
+% Each `?_` occurrence is anonymous and independent.
+materialize(answer, 1).
+pair(a, b).
+pair(c, d).
+answer(left(?x)) :- pair(?x, ?_).
+answer(right(?y)) :- pair(?_, ?y).

package/test/conformance/cases/variables/question_underscore_named_reuse.eye

@@ -0,0 +1,5 @@
+% `?_name` is a named variable and must be reused within a clause.
+materialize(answer, 1).
+pair(a, a).
+pair(a, b).
+answer(?_value) :- pair(?_value, ?_value).

package/test/conformance/errors/atoms/zero_arity_compound_rejected.eye

@@ -0,0 +1 @@
+p(nil()).

package/test/conformance/errors/syntax/colon_name_rejected.eye

@@ -0,0 +1 @@
+ex:alice.

package/test/conformance/errors/syntax/unclosed_quoted_atom_rejected.eye

@@ -0,0 +1 @@
+p('unterminated).

package/test/conformance/errors/variables/bare_underscore_rejected.eye

@@ -0,0 +1 @@
+p(_).

package/test/conformance/errors/variables/underscore_named_rejected.eye

@@ -0,0 +1 @@
+p(_name).

package/test/conformance/errors/variables/uppercase_variable_rejected.eye

@@ -0,0 +1 @@
+p(X).

package/test/conformance/expected/atoms/graphic_angle_atom_stays_graphic.eye

@@ -0,0 +1 @@
+answer(<=>).

package/test/conformance/expected/atoms/iri_atoms_in_lists.eye

@@ -0,0 +1 @@
+answer([<urn:example:a>, <urn:example:b>, <urn:example:c>]).

package/test/conformance/expected/atoms/iri_quoted_absolute_readback.eye

@@ -0,0 +1,2 @@
+answer(<https://example.org/alice>).
+answer(<urn:example:bob>).

package/test/conformance/expected/atoms/iri_quoted_and_angle_unify.eye

@@ -0,0 +1 @@
+answer(ok).

package/test/conformance/expected/atoms/zero_arity_atom_roundtrip.eye

@@ -0,0 +1,3 @@
+answer(name, nil).
+answer(args, []).
+answer(term, nil).

package/test/conformance/expected/declarations/memoize_is_ordinary_fact.eye

@@ -0,0 +1 @@
+answer(path, 2).

package/test/conformance/expected/declarations/mode_determinism_are_facts.eye

@@ -0,0 +1,3 @@
+answer(mode, [in, out]).
+answer(semidet, edge).
+answer(det, root).

package/test/conformance/expected/negation/stratified_negation_answers.eye

@@ -0,0 +1 @@
+open(a).

package/test/conformance/expected/table/table_does_not_change_answers.eye

@@ -0,0 +1,3 @@
+path(a, b).
+path(b, c).
+path(a, c).

package/test/conformance/expected/variables/question_anonymous_not_reused.eye

@@ -0,0 +1,4 @@
+answer(left(a)).
+answer(left(c)).
+answer(right(b)).
+answer(right(d)).

package/test/conformance/expected/variables/question_underscore_named_reuse.eye

@@ -0,0 +1 @@
+answer(a).

package/test/conformance/expected-errors/atoms/zero_arity_compound_rejected.txt

@@ -0,0 +1 @@
+parse line 1: zero-arity compound syntax is not supported; use atom "nil" for arity zero data

package/test/conformance/expected-errors/syntax/colon_name_rejected.txt

@@ -0,0 +1 @@
+colon names are not supported; use name or prefix_name

package/test/conformance/expected-errors/syntax/unclosed_quoted_atom_rejected.txt

@@ -0,0 +1 @@
+parse line 1: unterminated quoted term

package/test/conformance/expected-errors/variables/bare_underscore_rejected.txt

@@ -0,0 +1 @@
+parse line 1: bad character "_"

package/test/conformance/expected-errors/variables/underscore_named_rejected.txt

@@ -0,0 +1 @@
+parse line 1: bad character "_"

package/test/conformance/expected-errors/variables/uppercase_variable_rejected.txt

@@ -0,0 +1 @@
+parse line 1: bad character "X"

package/test/conformance/expected-warnings/negation/stratified_quiet.eye

@@ -0,0 +1 @@
+answer(ok).

package/test/conformance/expected-warnings/negation/unstratified_mutual.txt

@@ -0,0 +1,3 @@
+eyelang warning: unstratified negation
+  p/1 depends negatively on q/1
+  q/1 depends negatively on p/1

package/test/conformance/expected-warnings/negation/unstratified_self.txt

@@ -0,0 +1,2 @@
+eyelang warning: unstratified negation
+  p/1 depends negatively on p/1

package/test/conformance/warnings/negation/stratified_quiet.eye

@@ -0,0 +1,4 @@
+% Stratified negation emits no portability warning.
+materialize(answer, 1).
+candidate(a).
+answer(ok) :- candidate(a), not(blocked(a)).

package/test/conformance/warnings/negation/unstratified_mutual.eye

@@ -0,0 +1,5 @@
+% Warnings report unstratified negation without changing normal execution.
+materialize(answer, 1).
+p(a) :- not(q(a)).
+q(a) :- not(p(a)).
+answer(ok).

package/test/conformance/warnings/negation/unstratified_self.eye

@@ -0,0 +1,4 @@
+% A direct negative self-dependency is reported as unstratified.
+materialize(answer, 1).
+p(a) :- not(p(a)).
+answer(ok).

package/test/run-conformance.mjs

@@ -1,6 +1,6 @@
 #!/usr/bin/env node
 // Conformance test runner.
-// It executes cases in-process so the conformance corpus measures engine behavior instead of Node process startup.
+// It executes normal cases in-process so the conformance corpus measures engine behavior instead of Node process startup.
 import fs from 'node:fs';
 import path from 'node:path';
 import { spawnSync } from 'node:child_process';
@@ -9,29 +9,67 @@ import { fileURLToPath } from 'node:url';
 import { TestReporter, isMainModule } from './test-style.mjs';
 
 const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)));
+const packageRoot = path.resolve(root, '..');
+const cliBin = path.join(packageRoot, 'src', 'bin.js');
 const filterArg = process.argv[2] ?? null;
 
 export function runConformance(reporter = new TestReporter(), requestedFilter = null) {
   const filter = requestedFilter ?? filterArg;
   const label = filter == null ? 'eyelang' : `eyelang ${filter}`;
   reporter.section(`Conformance ${label}`);
-  for (const file of listCaseFiles(filter)) runCaseFile(reporter, file);
+  for (const file of listCaseFiles('cases', filter)) runCaseFile(reporter, file);
+  for (const file of listCaseFiles('errors', filter)) runErrorFile(reporter, file);
+  for (const file of listCaseFiles('warnings', filter)) runWarningFile(reporter, file);
   reporter.sectionTotal(`conformance ${label}`);
 }
 
-function listCaseFiles(filter = null) {
-  const casesDir = path.join(root, 'conformance', 'cases');
-  return fs.readdirSync(casesDir)
-    .filter((name) => name.endsWith('.eye'))
-    .filter((name) => filter == null || name.includes(filter) || name.slice(0, -4) === filter)
+function listCaseFiles(kind, filter = null) {
+  const base = path.join(root, 'conformance', kind);
+  if (!fs.existsSync(base)) return [];
+  return listEyeFiles(base)
+    .filter((name) => matchesFilter(kind, name, filter))
     .sort();
 }
 
+function matchesFilter(kind, name, filter) {
+  if (filter == null) return true;
+  const stem = name.slice(0, -4);
+  const label = kind === 'errors' ? 'error' : kind === 'warnings' ? 'warning' : kind;
+  return name.includes(filter)
+    || stem === filter
+    || stem.includes(filter)
+    || `${kind}/${stem}`.includes(filter)
+    || `${label}/${stem}`.includes(filter);
+}
+
+function listEyeFiles(base, dir = base) {
+  const files = [];
+  for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
+    const full = path.join(dir, entry.name);
+    if (entry.isDirectory()) {
+      files.push(...listEyeFiles(base, full));
+    } else if (entry.isFile() && entry.name.endsWith('.eye')) {
+      files.push(path.relative(base, full).split(path.sep).join('/'));
+    }
+  }
+  return files;
+}
+
 function runCaseFile(reporter, file) {
   const name = file.slice(0, -4);
   reporter.test(name, () => runCase(name, file));
 }
 
+function runErrorFile(reporter, file) {
+  const name = file.slice(0, -4);
+  reporter.test(`error/${name}`, () => runErrorCase(name, file));
+}
+
+function runWarningFile(reporter, file) {
+  const name = file.slice(0, -4);
+  reporter.test(`warning/${name}`, () => runWarningCase(name, file));
+}
+
 function runCase(name, file) {
   const casesDir = path.join(root, 'conformance', 'cases');
   const expectedDir = path.join(root, 'conformance', 'expected');
@@ -41,13 +79,57 @@ function runCase(name, file) {
   const program = Program.parseSources([{ text, filename: file }], { sourceMetadata: false, markRecursive: false });
   const actual = run(program).stdout;
 
+  compareExpectedFile(expected, actual, name, 'output');
+}
+
+function runErrorCase(name, file) {
+  const casesDir = path.join(root, 'conformance', 'errors');
+  const expectedDir = path.join(root, 'conformance', 'expected-errors');
+  const programFile = path.join(casesDir, file);
+  const expected = path.join(expectedDir, `${name}.txt`);
+  const text = fs.readFileSync(programFile, 'utf8');
+  let actual = null;
+
+  try {
+    const program = Program.parseSources([{ text, filename: file }], { sourceMetadata: false, markRecursive: false });
+    run(program);
+  } catch (error) {
+    actual = `${error?.message ?? String(error)}\n`;
+  }
+
+  if (actual == null) throw new Error(`expected error for ${name}, but program succeeded`);
+  compareExpectedFile(expected, actual, name, 'error');
+}
+
+function runWarningCase(name, file) {
+  const warningsDir = path.join(root, 'conformance', 'warnings');
+  const expectedDir = path.join(root, 'conformance', 'expected-warnings');
+  const programFile = path.join(warningsDir, file);
+  const expectedStdout = path.join(expectedDir, `${name}.eye`);
+  const expectedStderr = path.join(expectedDir, `${name}.txt`);
+  const text = fs.readFileSync(programFile, 'utf8');
+  const result = spawnSync(process.execPath, [cliBin, '--warnings', '-'], {
+    cwd: packageRoot,
+    input: text,
+    encoding: 'utf8',
+  });
+
+  if (result.status !== 0) {
+    throw new Error(`warning case ${name} exited with ${result.status}\n${result.stderr}`.trimEnd());
+  }
+
+  compareExpectedFile(expectedStdout, result.stdout, name, 'warning stdout');
+  compareExpectedFile(expectedStderr, result.stderr, name, 'warning stderr');
+}
+
+function compareExpectedFile(expected, actual, name, kind) {
   if (!fs.existsSync(expected)) {
-    throw new Error(`missing expected file: ${path.relative(root, expected)}`);
+    throw new Error(`missing expected ${kind} file: ${path.relative(root, expected)}`);
   }
 
   const expectedText = fs.readFileSync(expected, 'utf8');
   if (expectedText !== actual) {
-    throw new Error(`output mismatch for ${name}\n${diffText(expected, actual)}`.trimEnd());
+    throw new Error(`${kind} mismatch for ${name}\n${diffText(expected, actual)}`.trimEnd());
   }
 }