@interf/compiler 0.2.4 → 0.3.0

package/README.md

@@ -1,33 +1,19 @@
 # Interf
 
-Interf Knowledge Compiler uses local agents such as Claude Code and Codex to run a data-processing workflow over your files.
+Open-source knowledge compiler for your files.
 
-It creates a workspace with notes and navigation so the agent can see what is in the folder and what to retrieve.
+Interf measures and improves how accurately local agents answer questions from your files.
 
-Then you test that workspace on your evals.
+If you use Claude Code, Codex, OpenClaw, Hermes, or your own local agent setup on folders full of PDFs, docs, spreadsheets, and notes, the failure often shows up late: missed evidence, shallow analysis, bad comparisons, or answers that sound confident but are wrong.
+
+Interf lets you write a few questions and expected answers about your files, build a compiled workspace on top of those files, and see whether the result actually passes.
 
 - your files stay on your machine
 - you choose the local agent
-- you decide what must be true
-
-Agents start missing things when a task spans PDFs, charts, and several files in one folder. That usually shows up when the job depends on:
-
-- reading reports and filings
-- extracting a number from a chart
-- understanding what is inside a folder before doing work
-- pulling context together across several files
-- checking the raw source when the answer has to be exact
-
-The workspace exists so the agent does not have to rediscover the folder from scratch on every run.
+- your raw files stay the source of truth
+- Interf adds a file-based layer on top
 
-The point is proof on your data, not generic AI claims.
-
-The simplest way to use Interf is to compare the same task before and after compilation:
-
-- run the task on the raw folder
-- compile the folder with Interf
-- run the same task again from the workspace
-- if you want a recorded pass/fail result, add evals and run `interf benchmark`
+It runs local data-processing pipelines with your agents as executors and produces a compiled workspace: a file-based layer on top of your raw files that agents can navigate, inspect, and work from.
 
 ## Quick Start
 
@@ -36,95 +22,104 @@ Requirements:
 - Node.js 20+
 - a local coding agent: Claude Code or Codex
 
-Install and check setup:
+Install:
 
 ```bash
 npm install -g @interf/compiler
-interf doctor
 ```
 
-Then run Interf in any folder:
+Start by writing a few questions and expected answers in `interf.config.json`:
+
+```json
+{
+  "workspaces": [
+    {
+      "name": "default",
+      "about": "General compiled workspace for the quarterly results folder.",
+      "checks": [
+        {
+          "question": "What full-year revenue range did the company maintain?",
+          "answer": "$4.8B to $5.0B in revenue."
+        },
+        {
+          "question": "Did gross margin improve or decline year over year?",
+          "answer": "Gross margin declined year over year."
+        }
+      ]
+    }
+  ]
+}
+```
+
+Then run Interf in that folder:
 
 ```bash
 cd ~/my-folder
 interf init
+interf create workspace
 interf compile
 interf benchmark
 ```
 
-That is the whole first loop:
+That first run gives you three concrete things:
+
+- `interf/workspaces/default/` with the compiled workspace for your files
+- `interf/benchmarks/runs/...` with the saved benchmark result
+- a pass/fail score on the same questions and expected answers you wrote
+
+If `interf.config.json` is missing, `interf init` can draft it with you before the first compile. If Interf cannot find your local agent or compile setup, run:
+
+```bash
+interf doctor
+```
+
+The first flow is:
+
+- write down a few questions your agent should be able to answer from your files
+- let `interf init` save those checks in `interf.config.json`
+- run `interf create workspace` and `interf compile` to build the compiled workspace
+- run `interf benchmark` to see whether that compiled workspace passes the checks
+- add another workspace only when one recurring context needs a narrower setup
+- if needed, rerun compile or use the advanced retry path until it is good enough
 
-- point Interf at a folder you already have
-- let `interf init` write the first evals in `interf.config.json`
-- compile the workspace
-- ask the agent to use it
-- run `interf benchmark` to compare raw vs compiled
+## Why This Approach
 
-`interf init` chooses your local agent, can draft `interf.config.json` if it is missing, and can attach the current folder right away. It does not move or replace your files.
+Interf is built around a few simple design principles:
 
-Fastest sample loop:
+- `Explicit`: the output is visible and inspectable, not hidden memory
+- `Local`: your files stay on your machine
+- `File over app`: the output is just files, so you can use your editor, Unix tools, Obsidian, or your own software on top
+- `BYOAI`: use Claude Code, Codex, OpenClaw, Hermes, or your own model
+
+Interf does not replace your data with an opaque store. It keeps the raw files in place and adds a file-based layer on top for agents.
+
+Sample flow:
 
 ```bash
 cp -r examples/benchmark-demo /tmp/interf-demo
 cd /tmp/interf-demo
 interf init
+interf create workspace
 interf compile
 interf benchmark
 ```
 
-If you want a task-specific workspace for one job, add an interface:
+## Start With A Few Questions
 
-```bash
-interf create interface
-interf compile
-interf benchmark
-```
+`interf.config.json` is where you write the questions and expected answers for a folder.
 
-## Start With One Small Eval
+That file uses one `workspaces` array:
 
-`interf.config.json` is where you write what must be true.
+- the first workspace is the default compiled workspace for the folder
+- later workspaces are optional and only exist when one recurring context needs a narrower setup
+- each workspace carries its own `checks`
 
 If the file is missing, `interf init` can draft it with you before the first compile. You can edit it any time.
 
-Use it for:
-
-- top-level `evals` for shared baseline checks
-- `interfaces[].evals` for task-specific additional checks
-
-Both live in the same root `interf.config.json`.
-
-Example shape:
-
-Top-level `evals` are shared baseline checks for the workspace and every interface. Each entry in `interfaces` adds extra checks for one dedicated job.
-
-```json
-{
-  "evals": [
-    {
-      "question": "What was Bristol annual take-up in 2018, in millions of square feet?",
-      "answer": "About 0.5 million square feet. Accept answers between 0.3 and 0.6 if they clearly refer to Bristol annual take-up in 2018."
-    }
-  ],
-  "interfaces": [
-    {
-      "name": "market-briefing",
-      "about": "Prepare a short briefing from the office market report.",
-      "evals": [
-        {
-          "question": "What was Bristol availability in 2018, in millions of square feet?",
-          "answer": "About 0.6 million square feet. Accept answers between 0.5 and 0.7 if they clearly refer to Bristol availability in 2018."
-        },
-        {
-          "question": "Did Bristol annual take-up rise or fall between 2016 and 2018?",
-          "answer": "It fell. The chart shows roughly 0.7 to 0.8 million square feet in 2016 and about 0.5 million square feet in 2018."
-        }
-      ]
-    }
-  ]
-}
-```
+That example is just `interf.config.json`.
+Advanced retry settings do not live there.
 
-Good first evals are small and practical:
+Good first checks are small and practical:
 
 - one exact number from a chart, table, or filing
 - one short statement that should be true or false
@@ -133,136 +128,187 @@ Good first evals are small and practical:
 Then run:
 
 ```bash
+interf create workspace
 interf compile
 interf benchmark
 ```
 
-If the benchmark does not show an improvement over raw files, keep iterating or move to the experiment loop described below.
+## What `interf benchmark` Compares
 
-## Compare Three Things
+`interf benchmark` compares compiled workspaces, not raw chat sessions.
 
-Compare:
+It lets you answer a simple question:
 
-1. the raw folder
-2. the workspace
-3. an interface for one specific job
+- does this compiled workspace pass the checks?
+- which compiled workspace or workflow performs better on the same folder?
+- does another workspace help more on a narrower recurring context?
 
-`interf benchmark` is how you compare those on the same evals.
+By default it loads checks from `interf.config.json`, discovers eligible compiled workspaces under `interf/workspaces/`, and saves the run under `interf/benchmarks/runs/`.
 
-That gives you one clear question:
+If you run `interf benchmark` from inside a workspace, it uses that workspace's checks. If you run it from the source folder, it uses the default workspace checks.
 
-- is the raw folder enough?
-- does the workspace retrieve better?
-- does a dedicated interface do better than both?
+If you need raw-file probes too, use the advanced eval-pack runner in [docs/eval-loop.md](./docs/eval-loop.md).
 
-## What `interf compile` Actually Does
+## What `interf compile` Does
 
-`interf compile` runs a workflow over your folder.
+`interf compile` runs the Interf data-processing pipeline over your files.
 
-That workflow is the compilation pipeline:
+By default, that means:
 
 - read the files
-- write processed notes and navigation files
-- build the workspace your agent can use
-- optionally build an interface for one specific job
+- write summaries and navigation docs
+- organize cross-file knowledge
+- build the compiled workspace for the folder
+- optionally build additional workspaces for recurring contexts you care about
+
+In public docs, `pipeline` is the thing Interf runs. `workflow` is the saved method that defines or customizes that pipeline.
 
 The default workflow is built in. If you want a different method, you can define your own workflow package and benchmark it on the same folder.
 
-## Experiment Loop
+## What Gets Created
 
-Interf Knowledge Compiler also supports an experiment loop above compile + benchmark.
+After compile, Interf writes into `./interf/` beside your source files.
 
-It runs controlled experiments against the same folder and the same evals. Each attempt reruns the compilation workflow, reruns the benchmark, and records what changed. It stops when:
+- `interf/workspaces/<name>/` is a compiled workspace over the folder
+- `interf/benchmarks/runs/...` stores saved benchmark runs
 
-- the evals pass
-- or the experiment budget is exhausted
+Inside those workspaces you will see things like:
 
-In practice, that means:
+- summaries of source files
+- navigation notes and entrypoints for agents
+- cross-file knowledge notes
+- workspace-specific outputs for one recurring context
+- benchmark artifacts you can inspect later
 
-- `retry_policy.max_attempts_per_profile` controls how many experiment attempts each compile profile gets
-- stronger diagnostic profiles can be used only after the default ones fail
-- the loop is still judged on the same eval truth from your folder
-- failure summaries can be captured between attempts for diagnosis
+The compiled workspace is just a normal folder. Open it in your editor, in your agent, or in Obsidian if you want the graph view.
 
-Today that advanced path is configured through eval packs and explained in the deeper docs. The workflow is the part you change. The experiment loop is the controller that runs those experiments against the same evals with a fixed attempt budget.
+If you use Obsidian, open `interf/workspaces/<name>/` as the vault for the compiled workspace.
 
-Use the simple loop first. Use the experiment loop when you want to test workflow or profile changes against the same evals until one passes or the attempt budget runs out.
+## Terminology
 
-## Use It With Your Agent
+Public terms:
 
-If you already work through Claude Code, Codex, OpenClaw, or Hermes, the agent can run this loop for you.
+- `your files` = the source folder Interf reads from
+- `questions and expected answers` = the checks you want your agent to pass
+- `checks` = the pass/fail questions each workspace should satisfy
+- `benchmark` = score compiled workspaces on the same checks
+- `compiled workspace` = the output Interf produces on top of a folder
+- `workspace` = an optional additional compiled workspace for one recurring context
 
-Paste something like this into Claude Code, Codex, OpenClaw, or Hermes:
+Technical terms:
 
-```text
-Install @interf/compiler, run `interf init` in this folder, and use the local agent executor.
+- `source folder` = the raw files Interf reads from
+- `workflow` = the saved method that defines or customizes the pipeline
+- `.interf/` = runtime state, proofs, and health artifacts
 
-If `interf.config.json` is missing, draft evals for what must be true for this task and ask me to confirm them.
+## Advanced: Additional Workspaces
 
-Then run `interf compile` and `interf benchmark`.
+Start with the default compiled workspace first.
 
-Tell me whether the processed workspace beat raw files, and only recommend it if it did.
-```
+Most folders only need one compiled workspace.
 
-That is the basic loop:
+Create another workspace only when one recurring context needs a narrower setup.
 
-- the user or agent defines what must be true
-- Interf prepares processed data for retrieval
-- the benchmark shows whether that helped
+Use the default compiled workspace for broad questions such as:
 
-## What Gets Created
+- what is in this folder?
+- what changed?
+- where is the source evidence?
 
-After compile, Interf writes into `./interf/` beside your source files.
+Create another workspace when that broad layer is no longer enough and the work becomes a repeatable context, for example:
 
-- `interf/<name>/` is the shared workspace over the folder
-- `interf/<name>/interfaces/<name>/` is a task-specific workspace for one job
-- `interf/benchmarks/runs/...` stores saved benchmark runs
+- operator briefings
+- board prep
+- finance reporting
+- diligence review
+- chart extraction for a report set
+- one recurring research set
 
-Inside those workspaces you will see things like:
+Why create one:
 
-- summaries of source files
-- navigation notes and entrypoints for agents
-- task-specific outputs for one interface
-- benchmark artifacts you can inspect later
+- it narrows what the agent should retrieve for that context
+- it prepares guidance and structure on top of the raw folder for that recurring context
+- it lets you benchmark that context separately
 
-In the CLI, the main Interf workspace is called a **knowledge base**. A task-specific workspace inside it is called an **interface**.
+If the default compiled workspace is already enough, do not create another workspace yet.
 
-## When To Create An Interface
+## Advanced: Keep Improving Until It Passes
 
-Start with one workspace.
+Interf also supports an advanced experiment path above the normal build + benchmark flow.
 
-Create an interface when your agent needs outputs shaped for one specific job, for example:
+Give it the same folder and the same checks. Interf can keep rerunning compile + benchmark attempts until the benchmark passes or the attempt budget runs out.
 
-- weekly briefing
-- diligence on a deal room
-- extracting chart values from research PDFs
-- a focused research assistant for one question set
+In practice:
 
-If that workspace is enough for the job, you do not need an interface yet.
+- `retry_policy.max_attempts_per_profile` controls how many attempts each compile profile gets
+- stronger diagnostic profiles can be used only after the default ones fail
+- the checks stay the same across every attempt
+- each attempt records what changed
+
+Example eval-pack shape:
+
+```jsonc
+{
+  "workspaces": [
+    {
+      "name": "default",
+      "checks": [
+        {
+          "question": "What full-year revenue range did the company maintain?",
+          "answer": "$4.8B to $5.0B in revenue."
+        }
+      ]
+    }
+  ],
+  // Advanced only: retry settings live in eval packs, not in interf.config.json.
+  "retry_policy": {
+    "max_attempts_per_profile": 3
+  }
+}
+```
+
+Today this lives in the advanced eval-pack runner, not in `interf.config.json` and not in a top-level `interf compile --max-retries` flag.
+
+Use the normal benchmark flow first. Use this advanced path when you want Interf to keep improving the local preparation workflow until the workspace is good enough for your task or the attempt budget runs out. It spends more tokens, so use it when that extra spend is worth the accuracy target.
+
+## Use It With Your Agent
+
+If you already work through Claude Code, Codex, OpenClaw, or Hermes, the agent can run this process for you.
+
+Paste something like this into your agent:
+
+```text
+Install @interf/compiler, run `interf init` in this folder, and use the local agent executor.
+
+If `interf.config.json` is missing, draft a default workspace with a few checks this agent should be able to answer from these files and add the expected answers for me to confirm.
+
+Then run `interf create workspace`, `interf compile`, and `interf benchmark`.
+
+Tell me whether the compiled workspace passes the checks, and only recommend it if it does.
+```
 
 ## Custom Workflows
 
 Interf ships with a default workflow.
 
-If you want to change how compilation happens on your data, this is the part you customize:
+If you want to change how the data-processing pipeline runs on your files, this is the part you customize:
 
 ```bash
 interf create workflow
 interf verify workflow --path <path>
 ```
 
-Then benchmark that workflow on the same folder and the same evals.
+Then benchmark that workflow on the same folder and the same checks.
 
 Workflow package docs live in [docs/workflow-spec.md](./docs/workflow-spec.md).
 
 ## Core Commands
 
-- `interf init` = choose your local executor and optionally attach the current folder
-- `interf create knowledge-base` = create the shared processed workspace for this folder
-- `interf create interface` = create a task-specific workspace on top
+- `interf init` = choose your local executor and draft checks
+- `interf create workspace` = create a compiled workspace for this folder
 - `interf create workflow` = create a reusable local workflow package
-- `interf compile` = build the current workspace
-- `interf benchmark` = compare raw files vs processed workspaces on your evals
+- `interf compile` = build the current compiled workspace
+- `interf benchmark` = score compiled workspaces on your checks
 - `interf doctor` = check local executor setup
 - `interf verify <check>` = run deterministic checks on major workflow steps
 - `interf reset <scope>` = remove generated state while keeping source files
@@ -272,7 +318,7 @@ Workflow package docs live in [docs/workflow-spec.md](./docs/workflow-spec.md).
 - [docs/workflow-spec.md](./docs/workflow-spec.md) for custom workflow packages
 - [docs/runtime-contract.md](./docs/runtime-contract.md) for the exact on-disk contract
 - [docs/architecture.md](./docs/architecture.md) for the deeper system model
-- [docs/eval-loop.md](./docs/eval-loop.md) for the advanced benchmark and experiment loop
+- [docs/eval-loop.md](./docs/eval-loop.md) for advanced eval-pack experiments across workflows and models
 
 Maintainers should use [CONTRIBUTING.md](./CONTRIBUTING.md) for test and release gates.
 

package/dist/commands/benchmark.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"benchmark.d.ts","sourceRoot":"","sources":["../../src/commands/benchmark.ts"],"names":[],"mappings":"AA0BA,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,OAAO,CAAC;AA2W3C,eAAO,MAAM,gBAAgB,EAAE,aA8D9B,CAAC"}
+{"version":3,"file":"benchmark.d.ts","sourceRoot":"","sources":["../../src/commands/benchmark.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,OAAO,CAAC;AA6B3C,eAAO,MAAM,gBAAgB,EAAE,aA2E9B,CAAC"}