openmerit 0.1.6-preview.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +13 -0
- package/README.md +45 -106
- package/dist/terminal/public/app.js +18 -18
- package/dist/terminal/public/styles.css +18 -0
- package/dist/terminal/src/cli.mjs +1 -2
- package/dist/terminal/src/collect.mjs +36 -14
- package/dist/terminal/src/repository-background.mjs +37 -0
- package/dist/terminal/src/repository-index.mjs +155 -62
- package/dist/terminal/src/repository-policy.mjs +3 -0
- package/dist/terminal/src/repository-worker.mjs +15 -0
- package/dist/terminal/src/server.mjs +28 -14
- package/docs/architecture.md +1 -1
- package/docs/commands.md +2 -2
- package/docs/index.md +12 -1
- package/docs/inference-terminal.md +32 -14
- package/docs/operations.md +1 -1
- package/docs/security.md +8 -4
- package/docs/site-faq-maintenance.md +22 -0
- package/docs/site-terminal-preview.md +13 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,18 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.6 — 2026-10-11
|
|
4
|
+
|
|
5
|
+
- Promote the code from `0.1.6-preview.5` to the default npm release. Install with `npm install -g openmerit`, then run `openmerit dash` from an application directory.
|
|
6
|
+
- Update the packaged README and terminal guides to present current capabilities and limits. The terminal is a starting point for OpenMerit; this release introduces no new routing or evaluation behavior.
|
|
7
|
+
- Retain the existing coordinator, protocol, Pi adapter, package exports, and Node.js requirement. Earlier npm versions remain available.
|
|
8
|
+
|
|
9
|
+
## 0.1.6-preview.5 — 2026-10-10
|
|
10
|
+
|
|
11
|
+
- Open the terminal before initial collection, then index repository declarations in a background worker with visible discovery, indexing, and linking progress. Replace repository-wide scan caps with continuing work batches.
|
|
12
|
+
- Reuse unchanged source/configuration metadata across scans, re-resolve changed dotenv bindings, and remove deleted declarations. Keep activity collection independent and disclose skipped files or interrupted indexing.
|
|
13
|
+
- Reuse unchanged reference catalogs in the browser and report complete issue counts alongside the bounded detail list. Request-history and runtime inventory limits are unchanged.
|
|
14
|
+
- Preserve ordinary function names containing `task_` instead of mistaking their suffix for a credential prefix.
|
|
15
|
+
|
|
3
16
|
## 0.1.6-preview.4 — 2026-10-10
|
|
4
17
|
|
|
5
18
|
- Index supported application code, nested configuration, custom endpoints, and conservative model/call-site references automatically during `openmerit dash`, with file-and-line evidence.
|
package/README.md
CHANGED
|
@@ -1,137 +1,76 @@
|
|
|
1
1
|
# OpenMerit
|
|
2
2
|
|
|
3
|
-
OpenMerit is
|
|
3
|
+
OpenMerit is being built to help applications make better inference decisions using evidence about their models, workloads, quality, cost, and latency.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Version **0.1.6** starts with visibility: the Inference Terminal gives you a local view of the inference your application declares and records. It is a starting point for OpenMerit, not the full product vision. This release brings the existing npm preview into the default installation.
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
and the protocol in one install. The Pi adapter is the first integration; other
|
|
9
|
-
harnesses can use the same core and protocol exports.
|
|
7
|
+
## Install and run
|
|
10
8
|
|
|
11
|
-
|
|
9
|
+
Requires **Node.js 22.19 or newer**. In your application's directory:
|
|
12
10
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
- one-run observations from rate, percentile, consistency, and reliability claims;
|
|
18
|
-
- missing evidence from poor performance;
|
|
19
|
-
- feasible candidates from dominated, unresolved, or ineligible candidates;
|
|
20
|
-
- an automatic nudge from an intelligent action performed by the harness;
|
|
21
|
-
- a recommendation from an authorized, verified model swap.
|
|
22
|
-
|
|
23
|
-
## Current capabilities
|
|
24
|
-
|
|
25
|
-
- Pi 0.87 extension with a single `/openmerit` command surface.
|
|
26
|
-
- Harness-inferred, user-confirmed task profiles with explicit metrics, sampling designs, constraints, and evaluation budgets.
|
|
27
|
-
- The complete initial quality, reliability, cost, latency, efficiency, and human-intervention metric catalogue.
|
|
28
|
-
- Setup-time preliminary model research in the source checkout, followed by baseline evidence collection, refreshed candidate discovery, and controlled challenger trials. The published `0.1.5` package still begins candidate discovery after the baseline.
|
|
29
|
-
- A normative, uncertainty-aware Pareto definition owned by OpenMerit.
|
|
30
|
-
- Harness-calculated frontier results independently checked by OpenMerit's conformance verifier.
|
|
31
|
-
- Versioned result schemas for all ten intents, reusable across harness adapters and exposed dynamically by Pi.
|
|
32
|
-
- Configurable swap approval, post-swap verification requests, and policy-based rollback requests. Automatic swaps can begin immediately if explicitly enabled with a zero verified-swap threshold.
|
|
33
|
-
- Project-local snapshots, redacted append-only audit events, evidence manifests, and optional best-effort event exporters under `.openmerit/`.
|
|
34
|
-
- Model-catalog change detection that can nudge the harness to reassess a mature baseline.
|
|
35
|
-
- User-confirmed task-count, elapsed-time, regression, catalogue-change, and post-swap check policies.
|
|
36
|
-
- Idempotent harness signals, durable pending work, native wakeup planning, and explicit scheduler capability gaps.
|
|
37
|
-
- In the unreleased source checkout, an opt-in macOS Pi LaunchAgent for time-based checks and a durable pause; published `0.1.5` still requires an external scheduler and has a partial, session-local pause.
|
|
38
|
-
|
|
39
|
-
## Responsibility boundary
|
|
40
|
-
|
|
41
|
-
| OpenMerit | Coding harness |
|
|
42
|
-
| --- | --- |
|
|
43
|
-
| Defines typed outcomes and Pareto conformance rules | Understands the product and user task |
|
|
44
|
-
| Tracks lifecycle, policy, budget, and evidence references | Builds evals and observability |
|
|
45
|
-
| Determines which approved nudge is due | Collects and calculates metrics |
|
|
46
|
-
| Verifies returned structure and frontier conformance | Discovers candidates and calculates the frontier |
|
|
47
|
-
| Selects intents according to confirmation and rollback policy | Applies and verifies an authorized model change |
|
|
48
|
-
|
|
49
|
-
Jev System One, when configured, belongs to the coding harness. Pi may use it aggressively for bounded ranking, triage, and fast judgment. OpenMerit contains no Jev client and accepts no Jev credential.
|
|
50
|
-
|
|
51
|
-
## Install with Pi
|
|
11
|
+
```sh
|
|
12
|
+
npm install -g openmerit
|
|
13
|
+
openmerit dash
|
|
14
|
+
```
|
|
52
15
|
|
|
53
|
-
|
|
16
|
+
The terminal opens in your browser while collection starts. It runs locally, bound to `127.0.0.1`, and prints its URL. Keep the command running while using it; press Ctrl+C to stop. Pi and model-provider credentials are not required for the terminal.
|
|
54
17
|
|
|
55
|
-
|
|
56
|
-
- Pi 0.87 with a supported model provider configured
|
|
18
|
+
You can also point it at a project explicitly:
|
|
57
19
|
|
|
58
20
|
```sh
|
|
59
|
-
|
|
60
|
-
pi list
|
|
21
|
+
openmerit dash /path/to/your/project
|
|
61
22
|
```
|
|
62
23
|
|
|
63
|
-
|
|
24
|
+
To update from an earlier release or `openmerit@preview`, run `npm install -g openmerit`, stop the old terminal process, and launch it again. To pin this release, use `npm install -g openmerit@0.1.6`. Remove it with `npm uninstall -g openmerit`.
|
|
64
25
|
|
|
65
|
-
|
|
66
|
-
project evidence under `.openmerit/` in the product repository. Version 0.1.5
|
|
67
|
-
replaces the 0.1.4 background watcher and standalone CLI with this
|
|
68
|
-
harness-driven lifecycle; it does not migrate the earlier home-directory state.
|
|
26
|
+
## What you can see today
|
|
69
27
|
|
|
70
|
-
|
|
28
|
+
- **Models and services:** application declarations from supported code and configuration, recorded provider connections, and supported project-linked runtime inventory.
|
|
29
|
+
- **Inference activity:** requests, tokens, latency, costs, and agent/task identifiers where supported application logs or explicit exports provide them.
|
|
30
|
+
- **Code and configuration evidence:** file-and-line references, indexing progress, unresolved references, and source coverage.
|
|
31
|
+
- **Compute and deployments:** explicitly linked hosts, model residency, and supplied hardware or capacity measurements, with their freshness and scope shown.
|
|
71
32
|
|
|
72
|
-
|
|
73
|
-
- `/openmerit setup` — explicitly rerun or recover automatic setup.
|
|
74
|
-
- `/openmerit assess` — check stored application metric windows against the baseline thresholds now (in the unreleased source checkout); start candidate discovery only when ready. Published `0.1.5` still asks Pi to assess.
|
|
75
|
-
- `/openmerit logs` — show the canonical audit, exporter-error, evidence, and state paths.
|
|
76
|
-
- `/openmerit frontier` — ask the harness to calculate a frontier now; OpenMerit verifies it.
|
|
77
|
-
- `/openmerit approve` — approve a verified proposal during supervised graduation.
|
|
78
|
-
- `/openmerit pause` and `/openmerit resume` — control some proactive nudges in the current Pi session; task-triggered checks can still run in 0.1.5. See [pause limitations](docs/commands.md#pause-and-resume).
|
|
79
|
-
- `/openmerit doctor` — show sanitized adapter diagnostics.
|
|
33
|
+
The terminal keeps declarations, runtime inventory, and recorded usage separate. Missing evidence stays unknown. Code alone cannot reconstruct request history, usage, costs, or running agents. Discovery supports specific source formats and syntax; it is not complete coverage of every application or provider.
|
|
80
34
|
|
|
81
|
-
##
|
|
35
|
+
## Local and read-only
|
|
82
36
|
|
|
83
|
-
|
|
37
|
+
The terminal reads supported project files, logs, exports, and authorized loopback runtime inventory endpoints. It does not execute your application code, call models, route inference, run evaluations, or change your model configuration. It does not retain prompts, responses, or credential values in its snapshots. Closing it leaves your workloads running.
|
|
84
38
|
|
|
85
|
-
|
|
39
|
+
Repository indexing runs in the background and reuses unchanged metadata in memory. Request history and runtime readers have documented limits. The local API can be reached by other processes or users on the same machine, so treat its model names, endpoint identifiers, and file paths as project data.
|
|
40
|
+
|
|
41
|
+
## Useful commands
|
|
86
42
|
|
|
87
43
|
```sh
|
|
88
|
-
|
|
89
|
-
openmerit dash
|
|
44
|
+
openmerit dash --no-open # Print the local URL without opening a browser
|
|
45
|
+
openmerit dash --json # Print one collected snapshot, then exit
|
|
46
|
+
openmerit dash --diagnostics # Print repository-index coverage counts only
|
|
47
|
+
openmerit dash --demo # Explore fictional data without local collection
|
|
48
|
+
openmerit dash --help
|
|
90
49
|
```
|
|
91
50
|
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
## Documentation
|
|
95
|
-
|
|
96
|
-
Read the searchable [OpenMerit documentation](https://openmerit.site/docs/). The Markdown sources below also ship with the package.
|
|
51
|
+
See the [Inference Terminal guide](docs/inference-terminal.md) for supported sources, setup, options, data handling, and limitations. You can also [explore the hosted fictional sample](https://dash.openmerit.site/) without installing anything; it does not inspect your environment.
|
|
97
52
|
|
|
98
|
-
|
|
99
|
-
- [Your first trial](docs/first-trial.md)
|
|
100
|
-
- [Command reference](docs/commands.md)
|
|
101
|
-
- [Budgets and permissions](docs/budgets.md)
|
|
102
|
-
- [Troubleshooting](docs/troubleshooting.md)
|
|
103
|
-
- [Architecture and responsibility boundary](docs/architecture.md)
|
|
104
|
-
- [Metrics and evidence](docs/metrics-and-evidence.md)
|
|
105
|
-
- [Improvement lifecycle](docs/lifecycle.md)
|
|
106
|
-
- [Harness-neutral automation](docs/automation.md)
|
|
107
|
-
- [Pi extension guide](docs/pi-extension.md)
|
|
108
|
-
- [Building another harness adapter](docs/adapter-guide.md)
|
|
109
|
-
- [Pareto conformance specification](docs/pareto-spec.md)
|
|
110
|
-
- [Project state and operations](docs/operations.md)
|
|
111
|
-
- [Security](docs/security.md)
|
|
112
|
-
- [Testing](docs/testing.md)
|
|
113
|
-
- [Current limitations and roadmap](docs/roadmap.md)
|
|
53
|
+
## Optional Pi integration
|
|
114
54
|
|
|
115
|
-
|
|
55
|
+
The package also retains the Pi extension, reusable coordinator, and protocol. These are separate from the terminal. Pi can gather application evidence and carry out user-authorized model comparisons and changes; OpenMerit coordinates typed outcomes and validates the returned structure and frontier evidence.
|
|
116
56
|
|
|
117
|
-
|
|
57
|
+
For Pi 0.87 with a supported model provider configured:
|
|
118
58
|
|
|
119
|
-
|
|
59
|
+
```sh
|
|
60
|
+
pi install npm:openmerit
|
|
61
|
+
```
|
|
120
62
|
|
|
121
|
-
|
|
122
|
-
- `packages/core` — adapter SDK, durable coordinator, pluggable persistence, policy, and frontier conformance verification.
|
|
123
|
-
- `packages/pi` — Pi extension that turns OpenMerit intents into harness work.
|
|
124
|
-
- `docs` — user, operator, architecture, and normative documentation.
|
|
63
|
+
Then use `/openmerit doctor` inside Pi. See [Pi setup](docs/getting-started.md), [commands](docs/commands.md), [budgets and permissions](docs/budgets.md), and [current limitations](docs/roadmap.md). The terminal itself does not perform this evaluation or routing work.
|
|
125
64
|
|
|
126
|
-
The
|
|
127
|
-
package exposes `openmerit/core`, `openmerit/protocol`, and `openmerit/pi`.
|
|
65
|
+
The existing `openmerit/core`, `openmerit/protocol`, and `openmerit/pi` exports remain available for integrations.
|
|
128
66
|
|
|
129
|
-
##
|
|
67
|
+
## Documentation
|
|
130
68
|
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
69
|
+
- [Inference Terminal](docs/inference-terminal.md)
|
|
70
|
+
- [Security and data handling](docs/security.md)
|
|
71
|
+
- [Project files and logs](docs/operations.md)
|
|
72
|
+
- [Architecture and integrations](docs/architecture.md)
|
|
73
|
+
- [Testing and validation](docs/testing.md)
|
|
74
|
+
- [Changelog](CHANGELOG.md)
|
|
136
75
|
|
|
137
|
-
|
|
76
|
+
The bundled Inference Terminal guide follows this package version. The separately deployed [documentation website](https://openmerit.site/docs/) may reflect a different release.
|