@interf/compiler 0.3.0 → 0.3.2

package/README.md

@@ -1,19 +1,21 @@
 # Interf
 
-Open-source knowledge compiler for your files.
+Open-source knowledge compiler for local agents.
 
 Interf measures and improves how accurately local agents answer questions from your files.
 
 If you use Claude Code, Codex, OpenClaw, Hermes, or your own local agent setup on folders full of PDFs, docs, spreadsheets, and notes, the failure often shows up late: missed evidence, shallow analysis, bad comparisons, or answers that sound confident but are wrong.
 
-Interf lets you write a few questions and expected answers about your files, build a compiled workspace on top of those files, and see whether the result actually passes.
+Interf lets you define a few checks over your files, measure the raw baseline first if you want it, compile a workspace on top of those files, and see whether the result actually passes.
 
 - your files stay on your machine
 - you choose the local agent
 - your raw files stay the source of truth
 - Interf adds a file-based layer on top
 
-It runs local data-processing pipelines with your agents as executors and produces a compiled workspace: a file-based layer on top of your raw files that agents can navigate, inspect, and work from.
+`interf compile` runs a local data-processing pipeline with your agents as executors and produces a compiled workspace: a file-based layer on top of your raw files that agents can navigate, inspect, and work from.
+
+In the advanced looped mode, Interf can keep rerunning that pipeline, testing the result, and trying improved preparation attempts until it finds the best-performing workspace within the attempt budget.
 
 ## Quick Start
 
@@ -28,7 +30,14 @@ Install:
 npm install -g @interf/compiler
 ```
 
-Start by writing a few questions and expected answers in `interf.config.json`:
+The quickest start is the wizard:
+
+```bash
+cd ~/my-folder
+interf
+```
+
+If you want to see the config shape first, this is what Interf writes:
 
 ```json
 {
@@ -36,6 +45,9 @@ Start by writing a few questions and expected answers in `interf.config.json`:
     {
       "name": "default",
       "about": "General compiled workspace for the quarterly results folder.",
+      "retry_policy": {
+        "max_attempts": 3
+      },
       "checks": [
         {
           "question": "What full-year revenue range did the company maintain?",
@@ -51,23 +63,36 @@ Start by writing a few questions and expected answers in `interf.config.json`:
 }
 ```
 
-Then run Interf in that folder:
+The root-level flow is:
 
 ```bash
-cd ~/my-folder
-interf init
-interf create workspace
+interf
 interf compile
-interf benchmark
+interf test
 ```
 
-That first run gives you three concrete things:
+The first guided run can:
+
+- save a few questions and expected answers for this folder
+- run a baseline test on the raw files
+- compile the workspace
+- optionally keep compiling and retesting until it passes or reaches the attempt limit
+- run the same test against the compiled workspace
+
+That gives you three concrete things:
 
 - `interf/workspaces/default/` with the compiled workspace for your files
-- `interf/benchmarks/runs/...` with the saved benchmark result
+- `interf/benchmarks/runs/...` with the saved test result
 - a pass/fail score on the same questions and expected answers you wrote
 
-If `interf.config.json` is missing, `interf init` can draft it with you before the first compile. If Interf cannot find your local agent or compile setup, run:
+Saved test runs keep the details you need later:
+
+- whether the run tested `raw`, `workspace`, or both
+- per-question pass/fail results
+- the saved run path under `interf/benchmarks/runs/...`
+- executor metadata such as agent, command, model, effort, and profile when available
+
+If `interf.config.json` is missing, `interf` or `interf init` can draft it with you before the first compile. If Interf cannot find your local agent or compile setup, run:
 
 ```bash
 interf doctor
@@ -76,10 +101,11 @@ interf doctor
 The first flow is:
 
 - write down a few questions your agent should be able to answer from your files
-- let `interf init` save those checks in `interf.config.json`
-- run `interf create workspace` and `interf compile` to build the compiled workspace
-- run `interf benchmark` to see whether that compiled workspace passes the checks
-- add another workspace only when one recurring context needs a narrower setup
+- let `interf` or `interf init` save those checks in `interf.config.json`
+- optionally run a baseline test on the raw files
+- run `interf compile` to build the compiled workspace
+- run `interf test` to test the raw files, the compiled workspace, or both
+- only create another workspace if you want a separate compiled setup with its own checks
 - if needed, rerun compile or use the advanced retry path until it is good enough
 
 ## Why This Approach
@@ -98,27 +124,26 @@ Sample flow:
 ```bash
 cp -r examples/benchmark-demo /tmp/interf-demo
 cd /tmp/interf-demo
-interf init
-interf create workspace
+interf
 interf compile
-interf benchmark
+interf test
 ```
 
-## Start With A Few Questions
+## Start With Your Own Checks
+
+Start with your own checks over the files: questions where you already know the correct answer from the dataset.
 
-`interf.config.json` is where you write the questions and expected answers for a folder.
+`interf.config.json` is where you save those checks for a folder.
 
 That file uses one `workspaces` array:
 
-- the first workspace is the default compiled workspace for the folder
-- later workspaces are optional and only exist when one recurring context needs a narrower setup
+- most folders only need one workspace
+- add another workspace only if you want a separate compiled setup with different checks
 - each workspace carries its own `checks`
+- each workspace can optionally carry `retry_policy.max_attempts` for the self-improving compile loop
 
 If the file is missing, `interf init` can draft it with you before the first compile. You can edit it any time.
 
-That example is just `interf.config.json`.
-Advanced retry settings do not live there.
-
 Good first checks are small and practical:
 
 - one exact number from a chart, table, or filing
@@ -128,26 +153,34 @@ Good first checks are small and practical:
 Then run:
 
 ```bash
-interf create workspace
 interf compile
-interf benchmark
+interf test
 ```
 
-## What `interf benchmark` Compares
+## What `interf test` Does
 
-`interf benchmark` compares compiled workspaces, not raw chat sessions.
+`interf test` scores either the raw files, a compiled workspace, or both on the same saved checks.
 
 It lets you answer a simple question:
 
-- does this compiled workspace pass the checks?
+- what is the current baseline on the raw files?
+- does this compiled workspace improve on that baseline?
 - which compiled workspace or workflow performs better on the same folder?
-- does another workspace help more on a narrower recurring context?
+- does a separate workspace with different checks work better for that job?
+
+By default it loads checks from `interf.config.json`, can run a raw baseline in an isolated raw-files sandbox, can test eligible compiled workspaces under `interf/workspaces/`, and saves the run under `interf/benchmarks/runs/`.
+
+Each saved run includes:
 
-By default it loads checks from `interf.config.json`, discovers eligible compiled workspaces under `interf/workspaces/`, and saves the run under `interf/benchmarks/runs/`.
+- the benchmark target and mode
+- per-question results and traces
+- the executor metadata for that run
 
-If you run `interf benchmark` from inside a workspace, it uses that workspace's checks. If you run it from the source folder, it uses the default workspace checks.
+If you run `interf test` from inside a workspace, it uses that workspace's checks and tests that workspace. If you run it from the source folder, it lets you choose a saved workspace and then choose raw files, the compiled workspace, or both.
 
-If you need raw-file probes too, use the advanced eval-pack runner in [docs/eval-loop.md](./docs/eval-loop.md).
+Live test runs use an isolated sandbox. For raw baselines, Interf gives the agent sanitized raw files only. For compiled-workspace tests, it gives the agent a copied workspace plus sanitized raw files. The source-folder control plane, `interf.config.json`, and saved test runs are not part of those sandboxes.
+
+If you need repeated isolated experiments across workflows or models, use the advanced eval-pack runner in [docs/eval-loop.md](./docs/eval-loop.md).
 
 ## What `interf compile` Does
 
@@ -155,30 +188,35 @@ If you need raw-file probes too, use the advanced eval-pack runner in [docs/eval
 
 By default, that means:
 
-- read the files
-- write summaries and navigation docs
-- organize cross-file knowledge
-- build the compiled workspace for the folder
-- optionally build additional workspaces for recurring contexts you care about
+- summarize the source files into per-file evidence notes
+- structure the cross-file knowledge layer into entities, claims, and indexes
+- shape the final workspace around its saved focus and questions
+
+In other words, the built-in workflow is:
+
+1. `summarize`
+2. `structure`
+3. `shape`
 
 In public docs, `pipeline` is the thing Interf runs. `workflow` is the saved method that defines or customizes that pipeline.
 
 The default workflow is built in. If you want a different method, you can define your own workflow package and benchmark it on the same folder.
 
+If a workspace has `retry_policy.max_attempts`, or if you run `interf compile --max-attempts <n>`, Interf can keep compiling, testing, and retrying until that workspace passes or reaches the attempt limit. If several attempts fail, Interf keeps the best-performing compiled workspace from that run.
+
 ## What Gets Created
 
 After compile, Interf writes into `./interf/` beside your source files.
 
 - `interf/workspaces/<name>/` is a compiled workspace over the folder
-- `interf/benchmarks/runs/...` stores saved benchmark runs
+- `interf/benchmarks/runs/...` stores saved test runs
 
 Inside those workspaces you will see things like:
 
 - summaries of source files
 - navigation notes and entrypoints for agents
 - cross-file knowledge notes
-- workspace-specific outputs for one recurring context
-- benchmark artifacts you can inspect later
+- workspace-specific outputs when you define a separate job-focused workspace
 
 The compiled workspace is just a normal folder. Open it in your editor, in your agent, or in Obsidian if you want the graph view.
 
@@ -191,59 +229,56 @@ Public terms:
 - `your files` = the source folder Interf reads from
 - `questions and expected answers` = the checks you want your agent to pass
 - `checks` = the pass/fail questions each workspace should satisfy
-- `benchmark` = score compiled workspaces on the same checks
+- `test` = run the saved questions and get a score
 - `compiled workspace` = the output Interf produces on top of a folder
-- `workspace` = an optional additional compiled workspace for one recurring context
+- `workspace` = one compiled setup with its own checks
 
 Technical terms:
 
 - `source folder` = the raw files Interf reads from
+- `benchmark` = the technical alias and saved-run layer behind `interf test`
 - `workflow` = the saved method that defines or customizes the pipeline
 - `.interf/` = runtime state, proofs, and health artifacts
 
-## Advanced: Additional Workspaces
-
-Start with the default compiled workspace first.
-
-Most folders only need one compiled workspace.
-
-Create another workspace only when one recurring context needs a narrower setup.
-
-Use the default compiled workspace for broad questions such as:
+## Advanced: Separate Workspaces
 
-- what is in this folder?
-- what changed?
-- where is the source evidence?
+Most folders only need one workspace.
 
-Create another workspace when that broad layer is no longer enough and the work becomes a repeatable context, for example:
+Create another only when you want a different compiled setup with different checks, for example:
 
-- operator briefings
-- board prep
+- general folder understanding
 - finance reporting
+- board prep
 - diligence review
-- chart extraction for a report set
-- one recurring research set
 
-Why create one:
+Why create another one:
 
-- it narrows what the agent should retrieve for that context
-- it prepares guidance and structure on top of the raw folder for that recurring context
-- it lets you benchmark that context separately
-
-If the default compiled workspace is already enough, do not create another workspace yet.
+- it keeps a separate set of questions and expected answers
+- it gives that job its own compiled output under `interf/workspaces/<name>/`
+- it lets you test that job separately
 
 ## Advanced: Keep Improving Until It Passes
 
-Interf also supports an advanced experiment path above the normal build + benchmark flow.
+Interf also supports a deeper loop above the normal compile + test flow.
+
+The normal workspace flow already supports `retry_policy.max_attempts` inside `interf.config.json` or `interf compile --max-attempts <n>`.
 
-Give it the same folder and the same checks. Interf can keep rerunning compile + benchmark attempts until the benchmark passes or the attempt budget runs out.
+Give it the same folder and the same checks. Interf can keep rerunning compile + test attempts until the test passes or the attempt budget runs out.
+
+That loop is the self-improving part of the product:
+
+- it reruns the local data-processing pipeline over the same files
+- it keeps the checks fixed, so the target does not move
+- it can vary the compile profile and follow-up diagnostics
+- it records which attempt performed best on the same saved test
 
 In practice:
 
-- `retry_policy.max_attempts_per_profile` controls how many attempts each compile profile gets
+- `retry_policy.max_attempts` controls how many total attempts a normal workspace compile gets
+- `retry_policy.max_attempts_per_profile` controls how many attempts each compile profile gets in eval packs
 - stronger diagnostic profiles can be used only after the default ones fail
 - the checks stay the same across every attempt
-- each attempt records what changed
+- each attempt records what changed and which attempt performed best
 
 Example eval-pack shape:
 
@@ -252,6 +287,9 @@ Example eval-pack shape:
   "workspaces": [
     {
       "name": "default",
+      "retry_policy": {
+        "max_attempts": 3
+      },
       "checks": [
         {
           "question": "What full-year revenue range did the company maintain?",
@@ -260,16 +298,13 @@ Example eval-pack shape:
       ]
     }
   ],
-  // Advanced only: retry settings live in eval packs, not in interf.config.json.
   "retry_policy": {
     "max_attempts_per_profile": 3
   }
 }
 ```
 
-Today this lives in the advanced eval-pack runner, not in `interf.config.json` and not in a top-level `interf compile --max-retries` flag.
-
-Use the normal benchmark flow first. Use this advanced path when you want Interf to keep improving the local preparation workflow until the workspace is good enough for your task or the attempt budget runs out. It spends more tokens, so use it when that extra spend is worth the accuracy target.
+Use the normal workspace retry loop first. Use the eval-pack path when you want Interf to compare multiple compile profiles, add diagnostics, or keep iterating in a more controlled experiment loop. It spends more tokens, so use it when that extra spend is worth the accuracy target.
 
 ## Use It With Your Agent
 
@@ -278,11 +313,11 @@ If you already work through Claude Code, Codex, OpenClaw, or Hermes, the agent c
 Paste something like this into your agent:
 
 ```text
-Install @interf/compiler, run `interf init` in this folder, and use the local agent executor.
+Install @interf/compiler, run `interf` in this folder, and use the local agent executor.
 
-If `interf.config.json` is missing, draft a default workspace with a few checks this agent should be able to answer from these files and add the expected answers for me to confirm.
+If `interf.config.json` is missing, draft one workspace with a few checks this agent should be able to answer from these files and add the expected answers for me to confirm.
 
-Then run `interf create workspace`, `interf compile`, and `interf benchmark`.
+Then run a raw baseline if helpful, compile the workspace, and run `interf test`.
 
 Tell me whether the compiled workspace passes the checks, and only recommend it if it does.
 ```
@@ -304,11 +339,13 @@ Workflow package docs live in [docs/workflow-spec.md](./docs/workflow-spec.md).
 
 ## Core Commands
 
-- `interf init` = choose your local executor and draft checks
-- `interf create workspace` = create a compiled workspace for this folder
+- `interf` = open the root-folder wizard
+- `interf init` = alias for the root-folder wizard
+- `interf create workspace` = create another compiled workspace when you need one
 - `interf create workflow` = create a reusable local workflow package
-- `interf compile` = build the current compiled workspace
-- `interf benchmark` = score compiled workspaces on your checks
+- `interf compile` = build a selected workspace for the current folder
+- `interf test` = test the raw files, a compiled workspace, or both on saved checks
+- `interf benchmark` = alias for `interf test`
 - `interf doctor` = check local executor setup
 - `interf verify <check>` = run deterministic checks on major workflow steps
 - `interf reset <scope>` = remove generated state while keeping source files

package/dist/bin.js

@@ -4,7 +4,7 @@ import { hideBin } from "yargs/helpers";
 import { initCommand } from "./commands/init.js";
 import { createCommand } from "./commands/create.js";
 import { compileCommand } from "./commands/compile.js";
-import { benchmarkCommand } from "./commands/benchmark.js";
+import { testCommand } from "./commands/test.js";
 import { doctorCommand } from "./commands/doctor.js";
 import { listCommand } from "./commands/list.js";
 import { statusCommand } from "./commands/status.js";
@@ -15,9 +15,9 @@ yargs(hideBin(process.argv))
     .scriptName("interf")
     .command(defaultCommand)
     .command(initCommand)
-    .command(createCommand)
     .command(compileCommand)
-    .command(benchmarkCommand)
+    .command(testCommand)
+    .command(createCommand)
     .command(doctorCommand)
     .command(listCommand)
     .command(statusCommand)

package/dist/bin.js.map

@@ -1 +1 @@
-{"version":3,"file":"bin.js","sourceRoot":"","sources":["../src/bin.ts"],"names":[],"mappings":";AACA,OAAO,KAAK,MAAM,OAAO,CAAC;AAC1B,OAAO,EAAE,OAAO,EAAE,MAAM,eAAe,CAAC;AACxC,OAAO,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AACjD,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AACrD,OAAO,EAAE,cAAc,EAAE,MAAM,uBAAuB,CAAC;AACvD,OAAO,EAAE,gBAAgB,EAAE,MAAM,yBAAyB,CAAC;AAC3D,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AACrD,OAAO,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AACjD,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AACrD,OAAO,EAAE,YAAY,EAAE,MAAM,qBAAqB,CAAC;AACnD,OAAO,EAAE,cAAc,EAAE,MAAM,uBAAuB,CAAC;AACvD,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AAErD,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;KACzB,UAAU,CAAC,QAAQ,CAAC;KACpB,OAAO,CAAC,cAAc,CAAC;KACvB,OAAO,CAAC,WAAW,CAAC;KACpB,OAAO,CAAC,aAAa,CAAC;KACtB,OAAO,CAAC,cAAc,CAAC;KACvB,OAAO,CAAC,gBAAgB,CAAC;KACzB,OAAO,CAAC,aAAa,CAAC;KACtB,OAAO,CAAC,WAAW,CAAC;KACpB,OAAO,CAAC,aAAa,CAAC;KACtB,OAAO,CAAC,aAAa,CAAC;KACtB,OAAO,CAAC,YAAY,CAAC;KACrB,MAAM,EAAE;KACR,IAAI,EAAE;KACN,OAAO,EAAE;KACT,KAAK,EAAE,CAAC"}
+{"version":3,"file":"bin.js","sourceRoot":"","sources":["../src/bin.ts"],"names":[],"mappings":";AACA,OAAO,KAAK,MAAM,OAAO,CAAC;AAC1B,OAAO,EAAE,OAAO,EAAE,MAAM,eAAe,CAAC;AACxC,OAAO,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AACjD,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AACrD,OAAO,EAAE,cAAc,EAAE,MAAM,uBAAuB,CAAC;AACvD,OAAO,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AACjD,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AACrD,OAAO,EAAE,WAAW,EAAE,MAAM,oBAAoB,CAAC;AACjD,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AACrD,OAAO,EAAE,YAAY,EAAE,MAAM,qBAAqB,CAAC;AACnD,OAAO,EAAE,cAAc,EAAE,MAAM,uBAAuB,CAAC;AACvD,OAAO,EAAE,aAAa,EAAE,MAAM,sBAAsB,CAAC;AAErD,KAAK,CAAC,OAAO,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;KACzB,UAAU,CAAC,QAAQ,CAAC;KACpB,OAAO,CAAC,cAAc,CAAC;KACvB,OAAO,CAAC,WAAW,CAAC;KACpB,OAAO,CAAC,cAAc,CAAC;KACvB,OAAO,CAAC,WAAW,CAAC;KACpB,OAAO,CAAC,aAAa,CAAC;KACtB,OAAO,CAAC,aAAa,CAAC;KACtB,OAAO,CAAC,WAAW,CAAC;KACpB,OAAO,CAAC,aAAa,CAAC;KACtB,OAAO,CAAC,aAAa,CAAC;KACtB,OAAO,CAAC,YAAY,CAAC;KACrB,MAAM,EAAE;KACR,IAAI,EAAE;KACN,OAAO,EAAE;KACT,KAAK,EAAE,CAAC"}

package/dist/commands/compile.d.ts

@@ -1,3 +1,14 @@
+import type { WorkflowExecutionProfile, WorkflowExecutor } from "../lib/executors.js";
+import type { SourceWorkspaceConfig } from "../lib/schema.js";
 import type { CommandModule } from "yargs";
 export declare const compileCommand: CommandModule;
+export declare function runCompileCommand(argv?: Record<string, unknown>): Promise<void>;
+export declare function runConfiguredWorkspaceCompile(options: {
+    executor: WorkflowExecutor;
+    workspacePath: string;
+    sourcePath: string;
+    workspaceConfig: SourceWorkspaceConfig | null;
+    executionProfile?: WorkflowExecutionProfile;
+    maxAttemptsOverride: number | null;
+}): Promise<boolean>;
 //# sourceMappingURL=compile.d.ts.map

package/dist/commands/compile.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"compile.d.ts","sourceRoot":"","sources":["../../src/commands/compile.ts"],"names":[],"mappings":"AAqBA,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,OAAO,CAAC;AAoB3C,eAAO,MAAM,cAAc,EAAE,aAuC5B,CAAC"}
+{"version":3,"file":"compile.d.ts","sourceRoot":"","sources":["../../src/commands/compile.ts"],"names":[],"mappings":"AAUA,OAAO,KAAK,EAAE,wBAAwB,EAAE,gBAAgB,EAAE,MAAM,qBAAqB,CAAC;AAMtF,OAAO,KAAK,EAAE,qBAAqB,EAAE,MAAM,kBAAkB,CAAC;AAM9D,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,OAAO,CAAC;AAa3C,eAAO,MAAM,cAAc,EAAE,aAa5B,CAAC;AAEF,wBAAsB,iBAAiB,CAAC,IAAI,GAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CA+EzF;AA6CD,wBAAsB,6BAA6B,CACjD,OAAO,EAAE;IACP,QAAQ,EAAE,gBAAgB,CAAC;IAC3B,aAAa,EAAE,MAAM,CAAC;IACtB,UAAU,EAAE,MAAM,CAAC;IACnB,eAAe,EAAE,qBAAqB,GAAG,IAAI,CAAC;IAC9C,gBAAgB,CAAC,EAAE,wBAAwB,CAAC;IAC5C,mBAAmB,EAAE,MAAM,GAAG,IAAI,CAAC;CACpC,GACA,OAAO,CAAC,OAAO,CAAC,CA+GlB"}