qlm-measure 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- qlm_measure-0.1.0/PKG-INFO +169 -0
- qlm_measure-0.1.0/README.md +152 -0
- qlm_measure-0.1.0/pyproject.toml +28 -0
- qlm_measure-0.1.0/qlm_measure/__init__.py +27 -0
- qlm_measure-0.1.0/qlm_measure/clients.py +79 -0
- qlm_measure-0.1.0/qlm_measure/schema.py +126 -0
- qlm_measure-0.1.0/qlm_measure/verifier.py +166 -0
- qlm_measure-0.1.0/qlm_measure.egg-info/PKG-INFO +169 -0
- qlm_measure-0.1.0/qlm_measure.egg-info/SOURCES.txt +10 -0
- qlm_measure-0.1.0/qlm_measure.egg-info/dependency_links.txt +1 -0
- qlm_measure-0.1.0/qlm_measure.egg-info/top_level.txt +1 -0
- qlm_measure-0.1.0/setup.cfg +4 -0
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: qlm-measure
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Open measurement SDK for education AI — evidence events, record verification, and dataset clients
|
|
5
|
+
Author: Quantum Learning Machines
|
|
6
|
+
License: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://play.quantumlearningmachines.com/developer
|
|
8
|
+
Project-URL: Repository, https://github.com/quantumkumar/qlm-measure
|
|
9
|
+
Keywords: education,measurement,evidence,learning-analytics,misconceptions
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Topic :: Education
|
|
14
|
+
Classifier: Topic :: Scientific/Engineering
|
|
15
|
+
Requires-Python: >=3.9
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
|
|
18
|
+
# @qlm/measure
|
|
19
|
+
|
|
20
|
+
Open measurement SDK for education AI — evidence events, record verification, and dataset clients.
|
|
21
|
+
|
|
22
|
+
**v0.1.0 · Apache-2.0 · [Documentation](https://play.quantumlearningmachines.com/developer) · [Manifesto](https://play.quantumlearningmachines.com/resources/open-measurement-layer)**
|
|
23
|
+
|
|
24
|
+
## Limitations (read first)
|
|
25
|
+
|
|
26
|
+
- This SDK is the **public interface layer**, not the measurement engine. The estimation service (Bayesian posteriors, IRT ability estimation, calibration) is QLM's hosted engine.
|
|
27
|
+
- The verifier checks **bookkeeping integrity**, not estimation correctness. It can tell you if a record was tampered with. It cannot tell you if the posteriors are accurate. That distinction is the design.
|
|
28
|
+
- Evidence event schemas are **v0.1** and may change in minor versions. Pin your dependency.
|
|
29
|
+
- The ontology client requires the QLM API to be reachable. No offline mode.
|
|
30
|
+
- Classifier runners require model weights downloaded separately from Hugging Face.
|
|
31
|
+
|
|
32
|
+
## What it is
|
|
33
|
+
|
|
34
|
+
1. **Typed evidence-event vocabulary** (`ObservationEvent`) — a schema any tool can emit to describe a student interaction, with scaffold level, depth, epistemic mode.
|
|
35
|
+
2. **Buffered telemetry emitter** (`MeasurementEmitter`) — sends events to the measurement service with sendBeacon fallback.
|
|
36
|
+
3. **Evidence record format + replay verifier** (`verifyRecord`, `replayToVersion`) — audits a record's hash-chain and posterior-chain integrity without containing any estimation mathematics.
|
|
37
|
+
4. **Dataset clients** (`OntologyClient`) — typed wrappers over the public misconception ontology, learning graph, and standards alignment APIs.
|
|
38
|
+
5. **Engine client** (`EngineClient`) — commercial boundary for the hosted measurement service.
|
|
39
|
+
|
|
40
|
+
## What it is NOT
|
|
41
|
+
|
|
42
|
+
- Not the estimation engine (that runs server-side at QLM)
|
|
43
|
+
- Not a standalone learning analytics system
|
|
44
|
+
- Not a replacement for the measurement service
|
|
45
|
+
- Not a classifier or scorer (those are separate open-weights releases)
|
|
46
|
+
|
|
47
|
+
## Install
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
npm install @qlm/measure
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Quickstart
|
|
54
|
+
|
|
55
|
+
### 1. Query the misconception ontology (no auth)
|
|
56
|
+
|
|
57
|
+
```typescript
|
|
58
|
+
import { OntologyClient } from "@qlm/measure/clients";
|
|
59
|
+
|
|
60
|
+
const client = new OntologyClient();
|
|
61
|
+
const misconceptions = await client.getMisconceptions("math");
|
|
62
|
+
console.log(`${misconceptions.length} math misconceptions`);
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
### 2. Emit measurement events
|
|
66
|
+
|
|
67
|
+
```typescript
|
|
68
|
+
import { MeasurementEmitter } from "@qlm/measure/emitter";
|
|
69
|
+
|
|
70
|
+
const emitter = new MeasurementEmitter({
|
|
71
|
+
baseUrl: "https://play.quantumlearningmachines.com",
|
|
72
|
+
product: "my-tool",
|
|
73
|
+
});
|
|
74
|
+
|
|
75
|
+
emitter.emit({
|
|
76
|
+
eventType: "answer_submitted",
|
|
77
|
+
eventCategory: "learning",
|
|
78
|
+
domain: "math",
|
|
79
|
+
topic: "fractions",
|
|
80
|
+
payload: { questionId: "q1", correct: true },
|
|
81
|
+
});
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### 3. Verify an evidence record (tamper detection)
|
|
85
|
+
|
|
86
|
+
```typescript
|
|
87
|
+
import { verifyRecord } from "@qlm/measure/verifier";
|
|
88
|
+
|
|
89
|
+
const result = verifyRecord(record);
|
|
90
|
+
if (!result.valid) {
|
|
91
|
+
for (const v of result.violations) {
|
|
92
|
+
console.log(`[${v.category}] v${v.version}: ${v.message}`);
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Schema Reference
|
|
98
|
+
|
|
99
|
+
### ObservationEvent (evidence input)
|
|
100
|
+
|
|
101
|
+
| Field | Type | Required | Description |
|
|
102
|
+
|-------|------|----------|-------------|
|
|
103
|
+
| studentId | string | yes | Pseudonymous ID (no PII) |
|
|
104
|
+
| sessionId | string | yes | Session identifier |
|
|
105
|
+
| source | string | yes | Source system |
|
|
106
|
+
| timestamp | string | yes | ISO 8601 |
|
|
107
|
+
| correct | boolean | yes | Was the response correct? |
|
|
108
|
+
| responseTimeMs | number | yes | Response time in ms |
|
|
109
|
+
| domain | string | yes | Subject domain |
|
|
110
|
+
| scaffoldType | ScaffoldType | no | What preceded this response |
|
|
111
|
+
| misconceptionId | string | no | Detected misconception ID |
|
|
112
|
+
| classifierConfidence | number | no | Classifier confidence (0-1) |
|
|
113
|
+
| skillId | string | no | Skill being assessed |
|
|
114
|
+
| score | number | no | Response score (0-1) |
|
|
115
|
+
| difficulty | number | no | Item difficulty (0-1) |
|
|
116
|
+
| epistemicMode | EpistemicMode | no | How student arrived at answer |
|
|
117
|
+
| ext | Record | no | Extension namespace |
|
|
118
|
+
|
|
119
|
+
### Enums
|
|
120
|
+
|
|
121
|
+
**ScaffoldType:** `none`, `socratic`, `probing`, `metacognitive`, `scaffolding`, `explain`, `hint`, `demonstrate`
|
|
122
|
+
|
|
123
|
+
**EpistemicMode:** `experience`, `inference`, `analogy`, `testimony`
|
|
124
|
+
|
|
125
|
+
**DepthLevel:** `surface`, `conceptual`, `transfer`, `integration`
|
|
126
|
+
|
|
127
|
+
**TriageResult:** `correct`, `slip`, `misconception`, `disengagement`, `ambiguous`
|
|
128
|
+
|
|
129
|
+
## Verifier
|
|
130
|
+
|
|
131
|
+
The verifier checks:
|
|
132
|
+
- Schema validity (required fields, correct types)
|
|
133
|
+
- Strictly monotonic version numbers
|
|
134
|
+
- Timestamp monotonicity (configurable tolerance)
|
|
135
|
+
- Hash-chain integrity (prevHash linkage, entryHash recompute)
|
|
136
|
+
- Posterior chain consistency (each prior = previous updated)
|
|
137
|
+
- Compaction-boundary integrity
|
|
138
|
+
- Enum validity
|
|
139
|
+
|
|
140
|
+
What it deliberately **cannot** check:
|
|
141
|
+
- Whether posteriors are correctly computed (that requires the engine)
|
|
142
|
+
- Whether evidential weights are appropriate (those are calibrated server-side)
|
|
143
|
+
- Whether triage classifications are accurate (those use rules not in this package)
|
|
144
|
+
|
|
145
|
+
This is the design: **verify instead of trust**.
|
|
146
|
+
|
|
147
|
+
## Privacy
|
|
148
|
+
|
|
149
|
+
- `studentId` MUST be a pseudonymous identifier. Never include names, emails, or other PII.
|
|
150
|
+
- Events are buffered locally and sent via sendBeacon/fetch. No cookies are set.
|
|
151
|
+
- The SDK does not store any data persistently.
|
|
152
|
+
|
|
153
|
+
## Versioning
|
|
154
|
+
|
|
155
|
+
- Semver from 0.1.0.
|
|
156
|
+
- Schema changes in minor versions (0.2, 0.3).
|
|
157
|
+
- Breaking changes in major versions (1.0).
|
|
158
|
+
- Pin your dependency version.
|
|
159
|
+
|
|
160
|
+
## Links
|
|
161
|
+
|
|
162
|
+
- [Developer portal](https://play.quantumlearningmachines.com/developer)
|
|
163
|
+
- [Open Measurement Manifesto](https://play.quantumlearningmachines.com/resources/open-measurement-layer)
|
|
164
|
+
- [Misconception ontology](https://play.quantumlearningmachines.com/developer)
|
|
165
|
+
- [Model cards on Hugging Face](https://huggingface.co/QuantumLearningMachines)
|
|
166
|
+
|
|
167
|
+
## License
|
|
168
|
+
|
|
169
|
+
Apache-2.0. See [LICENSE](./LICENSE).
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# @qlm/measure
|
|
2
|
+
|
|
3
|
+
Open measurement SDK for education AI — evidence events, record verification, and dataset clients.
|
|
4
|
+
|
|
5
|
+
**v0.1.0 · Apache-2.0 · [Documentation](https://play.quantumlearningmachines.com/developer) · [Manifesto](https://play.quantumlearningmachines.com/resources/open-measurement-layer)**
|
|
6
|
+
|
|
7
|
+
## Limitations (read first)
|
|
8
|
+
|
|
9
|
+
- This SDK is the **public interface layer**, not the measurement engine. The estimation service (Bayesian posteriors, IRT ability estimation, calibration) is QLM's hosted engine.
|
|
10
|
+
- The verifier checks **bookkeeping integrity**, not estimation correctness. It can tell you if a record was tampered with. It cannot tell you if the posteriors are accurate. That distinction is the design.
|
|
11
|
+
- Evidence event schemas are **v0.1** and may change in minor versions. Pin your dependency.
|
|
12
|
+
- The ontology client requires the QLM API to be reachable. No offline mode.
|
|
13
|
+
- Classifier runners require model weights downloaded separately from Hugging Face.
|
|
14
|
+
|
|
15
|
+
## What it is
|
|
16
|
+
|
|
17
|
+
1. **Typed evidence-event vocabulary** (`ObservationEvent`) — a schema any tool can emit to describe a student interaction, with scaffold level, depth, epistemic mode.
|
|
18
|
+
2. **Buffered telemetry emitter** (`MeasurementEmitter`) — sends events to the measurement service with sendBeacon fallback.
|
|
19
|
+
3. **Evidence record format + replay verifier** (`verifyRecord`, `replayToVersion`) — audits a record's hash-chain and posterior-chain integrity without containing any estimation mathematics.
|
|
20
|
+
4. **Dataset clients** (`OntologyClient`) — typed wrappers over the public misconception ontology, learning graph, and standards alignment APIs.
|
|
21
|
+
5. **Engine client** (`EngineClient`) — commercial boundary for the hosted measurement service.
|
|
22
|
+
|
|
23
|
+
## What it is NOT
|
|
24
|
+
|
|
25
|
+
- Not the estimation engine (that runs server-side at QLM)
|
|
26
|
+
- Not a standalone learning analytics system
|
|
27
|
+
- Not a replacement for the measurement service
|
|
28
|
+
- Not a classifier or scorer (those are separate open-weights releases)
|
|
29
|
+
|
|
30
|
+
## Install
|
|
31
|
+
|
|
32
|
+
```bash
|
|
33
|
+
npm install @qlm/measure
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
## Quickstart
|
|
37
|
+
|
|
38
|
+
### 1. Query the misconception ontology (no auth)
|
|
39
|
+
|
|
40
|
+
```typescript
|
|
41
|
+
import { OntologyClient } from "@qlm/measure/clients";
|
|
42
|
+
|
|
43
|
+
const client = new OntologyClient();
|
|
44
|
+
const misconceptions = await client.getMisconceptions("math");
|
|
45
|
+
console.log(`${misconceptions.length} math misconceptions`);
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
### 2. Emit measurement events
|
|
49
|
+
|
|
50
|
+
```typescript
|
|
51
|
+
import { MeasurementEmitter } from "@qlm/measure/emitter";
|
|
52
|
+
|
|
53
|
+
const emitter = new MeasurementEmitter({
|
|
54
|
+
baseUrl: "https://play.quantumlearningmachines.com",
|
|
55
|
+
product: "my-tool",
|
|
56
|
+
});
|
|
57
|
+
|
|
58
|
+
emitter.emit({
|
|
59
|
+
eventType: "answer_submitted",
|
|
60
|
+
eventCategory: "learning",
|
|
61
|
+
domain: "math",
|
|
62
|
+
topic: "fractions",
|
|
63
|
+
payload: { questionId: "q1", correct: true },
|
|
64
|
+
});
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
### 3. Verify an evidence record (tamper detection)
|
|
68
|
+
|
|
69
|
+
```typescript
|
|
70
|
+
import { verifyRecord } from "@qlm/measure/verifier";
|
|
71
|
+
|
|
72
|
+
const result = verifyRecord(record);
|
|
73
|
+
if (!result.valid) {
|
|
74
|
+
for (const v of result.violations) {
|
|
75
|
+
console.log(`[${v.category}] v${v.version}: ${v.message}`);
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
## Schema Reference
|
|
81
|
+
|
|
82
|
+
### ObservationEvent (evidence input)
|
|
83
|
+
|
|
84
|
+
| Field | Type | Required | Description |
|
|
85
|
+
|-------|------|----------|-------------|
|
|
86
|
+
| studentId | string | yes | Pseudonymous ID (no PII) |
|
|
87
|
+
| sessionId | string | yes | Session identifier |
|
|
88
|
+
| source | string | yes | Source system |
|
|
89
|
+
| timestamp | string | yes | ISO 8601 |
|
|
90
|
+
| correct | boolean | yes | Was the response correct? |
|
|
91
|
+
| responseTimeMs | number | yes | Response time in ms |
|
|
92
|
+
| domain | string | yes | Subject domain |
|
|
93
|
+
| scaffoldType | ScaffoldType | no | What preceded this response |
|
|
94
|
+
| misconceptionId | string | no | Detected misconception ID |
|
|
95
|
+
| classifierConfidence | number | no | Classifier confidence (0-1) |
|
|
96
|
+
| skillId | string | no | Skill being assessed |
|
|
97
|
+
| score | number | no | Response score (0-1) |
|
|
98
|
+
| difficulty | number | no | Item difficulty (0-1) |
|
|
99
|
+
| epistemicMode | EpistemicMode | no | How student arrived at answer |
|
|
100
|
+
| ext | Record | no | Extension namespace |
|
|
101
|
+
|
|
102
|
+
### Enums
|
|
103
|
+
|
|
104
|
+
**ScaffoldType:** `none`, `socratic`, `probing`, `metacognitive`, `scaffolding`, `explain`, `hint`, `demonstrate`
|
|
105
|
+
|
|
106
|
+
**EpistemicMode:** `experience`, `inference`, `analogy`, `testimony`
|
|
107
|
+
|
|
108
|
+
**DepthLevel:** `surface`, `conceptual`, `transfer`, `integration`
|
|
109
|
+
|
|
110
|
+
**TriageResult:** `correct`, `slip`, `misconception`, `disengagement`, `ambiguous`
|
|
111
|
+
|
|
112
|
+
## Verifier
|
|
113
|
+
|
|
114
|
+
The verifier checks:
|
|
115
|
+
- Schema validity (required fields, correct types)
|
|
116
|
+
- Strictly monotonic version numbers
|
|
117
|
+
- Timestamp monotonicity (configurable tolerance)
|
|
118
|
+
- Hash-chain integrity (prevHash linkage, entryHash recompute)
|
|
119
|
+
- Posterior chain consistency (each prior = previous updated)
|
|
120
|
+
- Compaction-boundary integrity
|
|
121
|
+
- Enum validity
|
|
122
|
+
|
|
123
|
+
What it deliberately **cannot** check:
|
|
124
|
+
- Whether posteriors are correctly computed (that requires the engine)
|
|
125
|
+
- Whether evidential weights are appropriate (those are calibrated server-side)
|
|
126
|
+
- Whether triage classifications are accurate (those use rules not in this package)
|
|
127
|
+
|
|
128
|
+
This is the design: **verify instead of trust**.
|
|
129
|
+
|
|
130
|
+
## Privacy
|
|
131
|
+
|
|
132
|
+
- `studentId` MUST be a pseudonymous identifier. Never include names, emails, or other PII.
|
|
133
|
+
- Events are buffered locally and sent via sendBeacon/fetch. No cookies are set.
|
|
134
|
+
- The SDK does not store any data persistently.
|
|
135
|
+
|
|
136
|
+
## Versioning
|
|
137
|
+
|
|
138
|
+
- Semver from 0.1.0.
|
|
139
|
+
- Schema changes in minor versions (0.2, 0.3).
|
|
140
|
+
- Breaking changes in major versions (1.0).
|
|
141
|
+
- Pin your dependency version.
|
|
142
|
+
|
|
143
|
+
## Links
|
|
144
|
+
|
|
145
|
+
- [Developer portal](https://play.quantumlearningmachines.com/developer)
|
|
146
|
+
- [Open Measurement Manifesto](https://play.quantumlearningmachines.com/resources/open-measurement-layer)
|
|
147
|
+
- [Misconception ontology](https://play.quantumlearningmachines.com/developer)
|
|
148
|
+
- [Model cards on Hugging Face](https://huggingface.co/QuantumLearningMachines)
|
|
149
|
+
|
|
150
|
+
## License
|
|
151
|
+
|
|
152
|
+
Apache-2.0. See [LICENSE](./LICENSE).
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=68.0", "wheel"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "qlm-measure"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "Open measurement SDK for education AI — evidence events, record verification, and dataset clients"
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
license = {text = "Apache-2.0"}
|
|
11
|
+
requires-python = ">=3.9"
|
|
12
|
+
authors = [{name = "Quantum Learning Machines"}]
|
|
13
|
+
keywords = ["education", "measurement", "evidence", "learning-analytics", "misconceptions"]
|
|
14
|
+
classifiers = [
|
|
15
|
+
"Development Status :: 3 - Alpha",
|
|
16
|
+
"License :: OSI Approved :: Apache Software License",
|
|
17
|
+
"Programming Language :: Python :: 3",
|
|
18
|
+
"Topic :: Education",
|
|
19
|
+
"Topic :: Scientific/Engineering",
|
|
20
|
+
]
|
|
21
|
+
|
|
22
|
+
[project.urls]
|
|
23
|
+
Homepage = "https://play.quantumlearningmachines.com/developer"
|
|
24
|
+
Repository = "https://github.com/quantumkumar/qlm-measure"
|
|
25
|
+
|
|
26
|
+
[tool.setuptools.packages.find]
|
|
27
|
+
where = ["."]
|
|
28
|
+
include = ["qlm_measure*"]
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
"""
|
|
2
|
+
qlm-measure — Open measurement SDK for education AI.
|
|
3
|
+
|
|
4
|
+
Python mirror of @qlm/measure (npm). Provides:
|
|
5
|
+
- Evidence event schema (ObservationEvent, MeasurementEvent)
|
|
6
|
+
- Evidence record format + verifier
|
|
7
|
+
- Dataset clients (OntologyClient)
|
|
8
|
+
- Engine client (EngineClient, commercial boundary)
|
|
9
|
+
|
|
10
|
+
Apache-2.0 · v0.1.0
|
|
11
|
+
"""
|
|
12
|
+
|
|
13
|
+
__version__ = "0.1.0"
|
|
14
|
+
|
|
15
|
+
from .schema import (
|
|
16
|
+
ObservationEvent,
|
|
17
|
+
MeasurementEvent,
|
|
18
|
+
EvidenceEntry,
|
|
19
|
+
EvidenceRecord,
|
|
20
|
+
ScaffoldType,
|
|
21
|
+
EpistemicMode,
|
|
22
|
+
DepthLevel,
|
|
23
|
+
TriageResult,
|
|
24
|
+
COMPACTION_RETENTION,
|
|
25
|
+
)
|
|
26
|
+
from .verifier import verify_record, replay_to_version, posterior_at_version
|
|
27
|
+
from .clients import OntologyClient, EngineClient
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Dataset and engine clients — Python mirror.
|
|
3
|
+
"""
|
|
4
|
+
|
|
5
|
+
from __future__ import annotations
|
|
6
|
+
import json
|
|
7
|
+
from typing import Any, Optional
|
|
8
|
+
from urllib.request import urlopen, Request
|
|
9
|
+
from urllib.parse import urlencode
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
class OntologyClient:
|
|
13
|
+
"""Typed wrapper over the QLM dataset export API. Zero-auth."""
|
|
14
|
+
|
|
15
|
+
def __init__(self, base_url: str = "https://play.quantumlearningmachines.com"):
|
|
16
|
+
self.base_url = base_url
|
|
17
|
+
|
|
18
|
+
def fetch_dataset(
|
|
19
|
+
self,
|
|
20
|
+
dataset: str,
|
|
21
|
+
domain: Optional[str] = None,
|
|
22
|
+
fmt: Optional[str] = None,
|
|
23
|
+
) -> Any:
|
|
24
|
+
params = {"dataset": dataset}
|
|
25
|
+
if domain:
|
|
26
|
+
params["domain"] = domain
|
|
27
|
+
if fmt:
|
|
28
|
+
params["format"] = fmt
|
|
29
|
+
|
|
30
|
+
url = f"{self.base_url}/api/labpath/dataset-export?{urlencode(params)}"
|
|
31
|
+
req = Request(url, headers={"Accept": "application/json"})
|
|
32
|
+
with urlopen(req) as resp:
|
|
33
|
+
return json.loads(resp.read().decode())
|
|
34
|
+
|
|
35
|
+
def get_misconceptions(self, domain: Optional[str] = None) -> list[dict]:
|
|
36
|
+
data = self.fetch_dataset("misconceptions", domain=domain)
|
|
37
|
+
return data.get("misconceptions", data) if isinstance(data, dict) else data
|
|
38
|
+
|
|
39
|
+
def get_learning_graph(self, domain: Optional[str] = None) -> list[dict]:
|
|
40
|
+
data = self.fetch_dataset("learning-graph", domain=domain)
|
|
41
|
+
return data.get("learningGraph", data) if isinstance(data, dict) else data
|
|
42
|
+
|
|
43
|
+
def get_standards(self, domain: Optional[str] = None) -> list[dict]:
|
|
44
|
+
data = self.fetch_dataset("standards", domain=domain)
|
|
45
|
+
return data.get("standardsAlignment", data) if isinstance(data, dict) else data
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
class EngineClient:
|
|
49
|
+
"""
|
|
50
|
+
Commercial boundary for the QLM measurement engine.
|
|
51
|
+
|
|
52
|
+
The estimation service is QLM's hosted engine. This client wraps
|
|
53
|
+
the API for submitting observations and retrieving state.
|
|
54
|
+
"""
|
|
55
|
+
|
|
56
|
+
def __init__(self, base_url: str, api_key: str):
|
|
57
|
+
self.base_url = base_url
|
|
58
|
+
self.api_key = api_key
|
|
59
|
+
|
|
60
|
+
def _request(self, method: str, path: str, body: Optional[dict] = None) -> dict:
|
|
61
|
+
url = f"{self.base_url}{path}"
|
|
62
|
+
data = json.dumps(body).encode() if body else None
|
|
63
|
+
req = Request(
|
|
64
|
+
url,
|
|
65
|
+
data=data,
|
|
66
|
+
method=method,
|
|
67
|
+
headers={
|
|
68
|
+
"Content-Type": "application/json",
|
|
69
|
+
"Authorization": f"Bearer {self.api_key}",
|
|
70
|
+
},
|
|
71
|
+
)
|
|
72
|
+
with urlopen(req) as resp:
|
|
73
|
+
return json.loads(resp.read().decode())
|
|
74
|
+
|
|
75
|
+
def update_student_model(self, observation: dict) -> dict:
|
|
76
|
+
return self._request("POST", "/api/student-model/update", observation)
|
|
77
|
+
|
|
78
|
+
def get_state(self, student_id: str) -> dict:
|
|
79
|
+
return self._request("GET", f"/api/student-model/state?studentId={student_id}")
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Public event schema — Python mirror of TypeScript types.
|
|
3
|
+
|
|
4
|
+
All types are TypedDict for structural typing without runtime overhead.
|
|
5
|
+
"""
|
|
6
|
+
|
|
7
|
+
from __future__ import annotations
|
|
8
|
+
from typing import TypedDict, Literal, Optional, Any
|
|
9
|
+
|
|
10
|
+
# ── Enums (as Literal unions) ─────────────────────────────────────────────────
|
|
11
|
+
|
|
12
|
+
ScaffoldType = Literal[
|
|
13
|
+
"none", "socratic", "probing", "metacognitive",
|
|
14
|
+
"scaffolding", "explain", "hint", "demonstrate",
|
|
15
|
+
]
|
|
16
|
+
|
|
17
|
+
EpistemicMode = Literal["experience", "inference", "analogy", "testimony"]
|
|
18
|
+
|
|
19
|
+
DepthLevel = Literal["surface", "conceptual", "transfer", "integration"]
|
|
20
|
+
|
|
21
|
+
TriageResult = Literal["correct", "slip", "misconception", "disengagement", "ambiguous"]
|
|
22
|
+
|
|
23
|
+
EventCategory = Literal["learning", "teaching", "assessment", "interaction", "system"]
|
|
24
|
+
|
|
25
|
+
COMPACTION_RETENTION = 500
|
|
26
|
+
|
|
27
|
+
# ── Internal -> Public mapping ────────────────────────────────────────────────
|
|
28
|
+
|
|
29
|
+
_EPISTEMIC_MAP = {
|
|
30
|
+
"direct": "experience",
|
|
31
|
+
"reason": "inference",
|
|
32
|
+
"compare": "analogy",
|
|
33
|
+
"told": "testimony",
|
|
34
|
+
"experience": "experience",
|
|
35
|
+
"inference": "inference",
|
|
36
|
+
"analogy": "analogy",
|
|
37
|
+
"testimony": "testimony",
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
|
|
41
|
+
def to_public_epistemic_mode(value: str) -> EpistemicMode:
|
|
42
|
+
return _EPISTEMIC_MAP.get(value, "testimony") # type: ignore[return-value]
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
# ── ObservationEvent ──────────────────────────────────────────────────────────
|
|
46
|
+
|
|
47
|
+
class ObservationEvent(TypedDict, total=False):
|
|
48
|
+
studentId: str
|
|
49
|
+
sessionId: str
|
|
50
|
+
source: str
|
|
51
|
+
timestamp: str
|
|
52
|
+
correct: bool
|
|
53
|
+
responseTimeMs: int
|
|
54
|
+
domain: str
|
|
55
|
+
scaffoldType: Optional[ScaffoldType]
|
|
56
|
+
misconceptionId: Optional[str]
|
|
57
|
+
classifierConfidence: Optional[float]
|
|
58
|
+
classifierLabel: Optional[str]
|
|
59
|
+
skillId: Optional[str]
|
|
60
|
+
score: Optional[float]
|
|
61
|
+
difficulty: Optional[float]
|
|
62
|
+
consecutiveErrors: Optional[int]
|
|
63
|
+
sessionErrorRate: Optional[float]
|
|
64
|
+
explanationQuality: Optional[float]
|
|
65
|
+
isTransferProblem: Optional[bool]
|
|
66
|
+
connectionsMade: Optional[int]
|
|
67
|
+
epistemicMode: Optional[EpistemicMode]
|
|
68
|
+
selfCorrected: Optional[bool]
|
|
69
|
+
ext: Optional[dict[str, Any]]
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
# ── MeasurementEvent ──────────────────────────────────────────────────────────
|
|
73
|
+
|
|
74
|
+
class MeasurementEvent(TypedDict, total=False):
|
|
75
|
+
eventType: str
|
|
76
|
+
eventCategory: EventCategory
|
|
77
|
+
sessionId: Optional[str]
|
|
78
|
+
classroomId: Optional[str]
|
|
79
|
+
domain: Optional[str]
|
|
80
|
+
topic: Optional[str]
|
|
81
|
+
mode: Optional[str]
|
|
82
|
+
difficulty: Optional[float]
|
|
83
|
+
payload: Optional[dict[str, Any]]
|
|
84
|
+
qualitySignal: Optional[float]
|
|
85
|
+
confidence: Optional[float]
|
|
86
|
+
durationMs: Optional[int]
|
|
87
|
+
parentEventId: Optional[str]
|
|
88
|
+
|
|
89
|
+
|
|
90
|
+
# ── Evidence Record ───────────────────────────────────────────────────────────
|
|
91
|
+
|
|
92
|
+
class NonInterventionDecision(TypedDict):
|
|
93
|
+
intervened: bool
|
|
94
|
+
confidence: float
|
|
95
|
+
|
|
96
|
+
|
|
97
|
+
class RedactedEvent(TypedDict):
|
|
98
|
+
redacted: bool
|
|
99
|
+
eventHash: str
|
|
100
|
+
|
|
101
|
+
|
|
102
|
+
class EvidenceEntry(TypedDict, total=False):
|
|
103
|
+
version: int
|
|
104
|
+
timestamp: str
|
|
105
|
+
event: ObservationEvent | RedactedEvent
|
|
106
|
+
scaffoldType: ScaffoldType
|
|
107
|
+
depthLevel: DepthLevel
|
|
108
|
+
epistemicMode: Optional[EpistemicMode]
|
|
109
|
+
triageResult: TriageResult
|
|
110
|
+
evidentialWeight: float
|
|
111
|
+
priorPosterior: float
|
|
112
|
+
updatedPosterior: float
|
|
113
|
+
nonInterventionDecision: Optional[NonInterventionDecision]
|
|
114
|
+
entryHash: str
|
|
115
|
+
prevHash: str
|
|
116
|
+
|
|
117
|
+
|
|
118
|
+
class CompactedBefore(TypedDict):
|
|
119
|
+
version: int
|
|
120
|
+
summaryHash: str
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
class EvidenceRecord(TypedDict, total=False):
|
|
124
|
+
studentScopeId: str
|
|
125
|
+
entries: list[EvidenceEntry]
|
|
126
|
+
compactedBefore: Optional[CompactedBefore]
|
|
@@ -0,0 +1,166 @@
|
|
|
1
|
+
"""
|
|
2
|
+
Evidence Record Verifier — Python mirror.
|
|
3
|
+
|
|
4
|
+
Checks bookkeeping integrity WITHOUT any estimation mathematics.
|
|
5
|
+
"""
|
|
6
|
+
|
|
7
|
+
from __future__ import annotations
|
|
8
|
+
import hashlib
|
|
9
|
+
import json
|
|
10
|
+
from dataclasses import dataclass, field
|
|
11
|
+
from typing import Optional
|
|
12
|
+
|
|
13
|
+
from .schema import EvidenceRecord, EvidenceEntry
|
|
14
|
+
|
|
15
|
+
VALID_SCAFFOLD_TYPES = {"none", "socratic", "probing", "metacognitive", "scaffolding", "explain", "hint", "demonstrate"}
|
|
16
|
+
VALID_DEPTH_LEVELS = {"surface", "conceptual", "transfer", "integration"}
|
|
17
|
+
VALID_EPISTEMIC_MODES = {"experience", "inference", "analogy", "testimony"}
|
|
18
|
+
VALID_TRIAGE_RESULTS = {"correct", "slip", "misconception", "disengagement", "ambiguous"}
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
@dataclass
|
|
22
|
+
class Violation:
|
|
23
|
+
version: int
|
|
24
|
+
category: str # schema | version | timestamp | hash | posterior | compaction | enum
|
|
25
|
+
message: str
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
@dataclass
|
|
29
|
+
class VerificationResult:
|
|
30
|
+
valid: bool
|
|
31
|
+
violations: list[Violation] = field(default_factory=list)
|
|
32
|
+
entries_checked: int = 0
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
def _canonicalize(obj: object) -> str:
|
|
36
|
+
"""Canonical JSON: sorted keys, no whitespace."""
|
|
37
|
+
return json.dumps(obj, sort_keys=True, separators=(",", ":"), default=str)
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
def _sha256(s: str) -> str:
|
|
41
|
+
return hashlib.sha256(s.encode()).hexdigest()
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
def _hash_entry(entry: EvidenceEntry) -> str:
|
|
45
|
+
hashable = {
|
|
46
|
+
"version": entry.get("version"),
|
|
47
|
+
"timestamp": entry.get("timestamp"),
|
|
48
|
+
"event": entry.get("event"),
|
|
49
|
+
"scaffoldType": entry.get("scaffoldType"),
|
|
50
|
+
"depthLevel": entry.get("depthLevel"),
|
|
51
|
+
"epistemicMode": entry.get("epistemicMode"),
|
|
52
|
+
"triageResult": entry.get("triageResult"),
|
|
53
|
+
"evidentialWeight": entry.get("evidentialWeight"),
|
|
54
|
+
"priorPosterior": entry.get("priorPosterior"),
|
|
55
|
+
"updatedPosterior": entry.get("updatedPosterior"),
|
|
56
|
+
"nonInterventionDecision": entry.get("nonInterventionDecision"),
|
|
57
|
+
}
|
|
58
|
+
return _sha256(_canonicalize(hashable))
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
def verify_record(
|
|
62
|
+
record: EvidenceRecord,
|
|
63
|
+
timestamp_tolerance_ms: int = 1000,
|
|
64
|
+
) -> VerificationResult:
|
|
65
|
+
"""Verify bookkeeping integrity of an evidence record."""
|
|
66
|
+
violations: list[Violation] = []
|
|
67
|
+
|
|
68
|
+
scope_id = record.get("studentScopeId", "")
|
|
69
|
+
if not scope_id:
|
|
70
|
+
violations.append(Violation(0, "schema", "Missing or empty studentScopeId"))
|
|
71
|
+
|
|
72
|
+
entries = record.get("entries", [])
|
|
73
|
+
if not isinstance(entries, list):
|
|
74
|
+
violations.append(Violation(0, "schema", "entries must be a list"))
|
|
75
|
+
return VerificationResult(valid=False, violations=violations, entries_checked=0)
|
|
76
|
+
|
|
77
|
+
compacted = record.get("compactedBefore")
|
|
78
|
+
prev_version = compacted["version"] if compacted else 0
|
|
79
|
+
prev_hash = compacted.get("summaryHash", "") if compacted else ""
|
|
80
|
+
last_updated: Optional[float] = None
|
|
81
|
+
|
|
82
|
+
from datetime import datetime
|
|
83
|
+
|
|
84
|
+
prev_ts_str = ""
|
|
85
|
+
for entry in entries:
|
|
86
|
+
v = entry.get("version", -1)
|
|
87
|
+
|
|
88
|
+
# Schema
|
|
89
|
+
if not isinstance(v, int) or v < 1:
|
|
90
|
+
violations.append(Violation(v, "schema", "version must be a positive integer"))
|
|
91
|
+
if not entry.get("timestamp"):
|
|
92
|
+
violations.append(Violation(v, "schema", "timestamp is required"))
|
|
93
|
+
if not isinstance(entry.get("evidentialWeight"), (int, float)):
|
|
94
|
+
violations.append(Violation(v, "schema", "evidentialWeight must be a number"))
|
|
95
|
+
if not isinstance(entry.get("priorPosterior"), (int, float)):
|
|
96
|
+
violations.append(Violation(v, "schema", "priorPosterior must be a number"))
|
|
97
|
+
if not isinstance(entry.get("updatedPosterior"), (int, float)):
|
|
98
|
+
violations.append(Violation(v, "schema", "updatedPosterior must be a number"))
|
|
99
|
+
|
|
100
|
+
# Monotonic version
|
|
101
|
+
if v <= prev_version:
|
|
102
|
+
violations.append(Violation(v, "version", f"Version {v} <= previous {prev_version}"))
|
|
103
|
+
|
|
104
|
+
# Timestamp monotonicity
|
|
105
|
+
ts_str = entry.get("timestamp", "")
|
|
106
|
+
if prev_ts_str and ts_str:
|
|
107
|
+
try:
|
|
108
|
+
prev_ms = datetime.fromisoformat(prev_ts_str.replace("Z", "+00:00")).timestamp() * 1000
|
|
109
|
+
curr_ms = datetime.fromisoformat(ts_str.replace("Z", "+00:00")).timestamp() * 1000
|
|
110
|
+
if curr_ms < prev_ms - timestamp_tolerance_ms:
|
|
111
|
+
violations.append(Violation(v, "timestamp", f"Timestamp goes backward"))
|
|
112
|
+
except (ValueError, TypeError):
|
|
113
|
+
pass
|
|
114
|
+
|
|
115
|
+
# Enum validation
|
|
116
|
+
st = entry.get("scaffoldType")
|
|
117
|
+
if st and st not in VALID_SCAFFOLD_TYPES:
|
|
118
|
+
violations.append(Violation(v, "enum", f"Invalid scaffoldType: {st}"))
|
|
119
|
+
dl = entry.get("depthLevel")
|
|
120
|
+
if dl and dl not in VALID_DEPTH_LEVELS:
|
|
121
|
+
violations.append(Violation(v, "enum", f"Invalid depthLevel: {dl}"))
|
|
122
|
+
em = entry.get("epistemicMode")
|
|
123
|
+
if em and em not in VALID_EPISTEMIC_MODES:
|
|
124
|
+
violations.append(Violation(v, "enum", f"Invalid epistemicMode: {em}"))
|
|
125
|
+
tr = entry.get("triageResult")
|
|
126
|
+
if tr and tr not in VALID_TRIAGE_RESULTS:
|
|
127
|
+
violations.append(Violation(v, "enum", f"Invalid triageResult: {tr}"))
|
|
128
|
+
|
|
129
|
+
# Hash chain
|
|
130
|
+
if entry.get("prevHash") != prev_hash:
|
|
131
|
+
violations.append(Violation(v, "hash", f"prevHash mismatch"))
|
|
132
|
+
computed = _hash_entry(entry)
|
|
133
|
+
if entry.get("entryHash") != computed:
|
|
134
|
+
violations.append(Violation(v, "hash", f"entryHash mismatch"))
|
|
135
|
+
|
|
136
|
+
# Posterior chain
|
|
137
|
+
if last_updated is not None:
|
|
138
|
+
prior = entry.get("priorPosterior")
|
|
139
|
+
if isinstance(prior, (int, float)) and abs(prior - last_updated) > 1e-9:
|
|
140
|
+
violations.append(Violation(v, "posterior", f"priorPosterior ({prior}) != previous updatedPosterior ({last_updated})"))
|
|
141
|
+
|
|
142
|
+
prev_version = v
|
|
143
|
+
prev_ts_str = ts_str
|
|
144
|
+
prev_hash = entry.get("entryHash", "")
|
|
145
|
+
up = entry.get("updatedPosterior")
|
|
146
|
+
if isinstance(up, (int, float)):
|
|
147
|
+
last_updated = up
|
|
148
|
+
|
|
149
|
+
return VerificationResult(
|
|
150
|
+
valid=len(violations) == 0,
|
|
151
|
+
violations=violations,
|
|
152
|
+
entries_checked=len(entries),
|
|
153
|
+
)
|
|
154
|
+
|
|
155
|
+
|
|
156
|
+
def replay_to_version(record: EvidenceRecord, target_version: int) -> list[EvidenceEntry]:
|
|
157
|
+
"""Return entries up to and including target_version."""
|
|
158
|
+
return [e for e in record.get("entries", []) if e.get("version", 0) <= target_version]
|
|
159
|
+
|
|
160
|
+
|
|
161
|
+
def posterior_at_version(record: EvidenceRecord, target_version: int) -> Optional[float]:
|
|
162
|
+
"""Get the posterior at a specific version."""
|
|
163
|
+
entries = replay_to_version(record, target_version)
|
|
164
|
+
if not entries:
|
|
165
|
+
return None
|
|
166
|
+
return entries[-1].get("updatedPosterior")
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: qlm-measure
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Open measurement SDK for education AI — evidence events, record verification, and dataset clients
|
|
5
|
+
Author: Quantum Learning Machines
|
|
6
|
+
License: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://play.quantumlearningmachines.com/developer
|
|
8
|
+
Project-URL: Repository, https://github.com/quantumkumar/qlm-measure
|
|
9
|
+
Keywords: education,measurement,evidence,learning-analytics,misconceptions
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: License :: OSI Approved :: Apache Software License
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Topic :: Education
|
|
14
|
+
Classifier: Topic :: Scientific/Engineering
|
|
15
|
+
Requires-Python: >=3.9
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
|
|
18
|
+
# @qlm/measure
|
|
19
|
+
|
|
20
|
+
Open measurement SDK for education AI — evidence events, record verification, and dataset clients.
|
|
21
|
+
|
|
22
|
+
**v0.1.0 · Apache-2.0 · [Documentation](https://play.quantumlearningmachines.com/developer) · [Manifesto](https://play.quantumlearningmachines.com/resources/open-measurement-layer)**
|
|
23
|
+
|
|
24
|
+
## Limitations (read first)
|
|
25
|
+
|
|
26
|
+
- This SDK is the **public interface layer**, not the measurement engine. The estimation service (Bayesian posteriors, IRT ability estimation, calibration) is QLM's hosted engine.
|
|
27
|
+
- The verifier checks **bookkeeping integrity**, not estimation correctness. It can tell you if a record was tampered with. It cannot tell you if the posteriors are accurate. That distinction is the design.
|
|
28
|
+
- Evidence event schemas are **v0.1** and may change in minor versions. Pin your dependency.
|
|
29
|
+
- The ontology client requires the QLM API to be reachable. No offline mode.
|
|
30
|
+
- Classifier runners require model weights downloaded separately from Hugging Face.
|
|
31
|
+
|
|
32
|
+
## What it is
|
|
33
|
+
|
|
34
|
+
1. **Typed evidence-event vocabulary** (`ObservationEvent`) — a schema any tool can emit to describe a student interaction, with scaffold level, depth, epistemic mode.
|
|
35
|
+
2. **Buffered telemetry emitter** (`MeasurementEmitter`) — sends events to the measurement service with sendBeacon fallback.
|
|
36
|
+
3. **Evidence record format + replay verifier** (`verifyRecord`, `replayToVersion`) — audits a record's hash-chain and posterior-chain integrity without containing any estimation mathematics.
|
|
37
|
+
4. **Dataset clients** (`OntologyClient`) — typed wrappers over the public misconception ontology, learning graph, and standards alignment APIs.
|
|
38
|
+
5. **Engine client** (`EngineClient`) — commercial boundary for the hosted measurement service.
|
|
39
|
+
|
|
40
|
+
## What it is NOT
|
|
41
|
+
|
|
42
|
+
- Not the estimation engine (that runs server-side at QLM)
|
|
43
|
+
- Not a standalone learning analytics system
|
|
44
|
+
- Not a replacement for the measurement service
|
|
45
|
+
- Not a classifier or scorer (those are separate open-weights releases)
|
|
46
|
+
|
|
47
|
+
## Install
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
npm install @qlm/measure
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Quickstart
|
|
54
|
+
|
|
55
|
+
### 1. Query the misconception ontology (no auth)
|
|
56
|
+
|
|
57
|
+
```typescript
|
|
58
|
+
import { OntologyClient } from "@qlm/measure/clients";
|
|
59
|
+
|
|
60
|
+
const client = new OntologyClient();
|
|
61
|
+
const misconceptions = await client.getMisconceptions("math");
|
|
62
|
+
console.log(`${misconceptions.length} math misconceptions`);
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
### 2. Emit measurement events
|
|
66
|
+
|
|
67
|
+
```typescript
|
|
68
|
+
import { MeasurementEmitter } from "@qlm/measure/emitter";
|
|
69
|
+
|
|
70
|
+
const emitter = new MeasurementEmitter({
|
|
71
|
+
baseUrl: "https://play.quantumlearningmachines.com",
|
|
72
|
+
product: "my-tool",
|
|
73
|
+
});
|
|
74
|
+
|
|
75
|
+
emitter.emit({
|
|
76
|
+
eventType: "answer_submitted",
|
|
77
|
+
eventCategory: "learning",
|
|
78
|
+
domain: "math",
|
|
79
|
+
topic: "fractions",
|
|
80
|
+
payload: { questionId: "q1", correct: true },
|
|
81
|
+
});
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
### 3. Verify an evidence record (tamper detection)
|
|
85
|
+
|
|
86
|
+
```typescript
|
|
87
|
+
import { verifyRecord } from "@qlm/measure/verifier";
|
|
88
|
+
|
|
89
|
+
const result = verifyRecord(record);
|
|
90
|
+
if (!result.valid) {
|
|
91
|
+
for (const v of result.violations) {
|
|
92
|
+
console.log(`[${v.category}] v${v.version}: ${v.message}`);
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Schema Reference
|
|
98
|
+
|
|
99
|
+
### ObservationEvent (evidence input)
|
|
100
|
+
|
|
101
|
+
| Field | Type | Required | Description |
|
|
102
|
+
|-------|------|----------|-------------|
|
|
103
|
+
| studentId | string | yes | Pseudonymous ID (no PII) |
|
|
104
|
+
| sessionId | string | yes | Session identifier |
|
|
105
|
+
| source | string | yes | Source system |
|
|
106
|
+
| timestamp | string | yes | ISO 8601 |
|
|
107
|
+
| correct | boolean | yes | Was the response correct? |
|
|
108
|
+
| responseTimeMs | number | yes | Response time in ms |
|
|
109
|
+
| domain | string | yes | Subject domain |
|
|
110
|
+
| scaffoldType | ScaffoldType | no | What preceded this response |
|
|
111
|
+
| misconceptionId | string | no | Detected misconception ID |
|
|
112
|
+
| classifierConfidence | number | no | Classifier confidence (0-1) |
|
|
113
|
+
| skillId | string | no | Skill being assessed |
|
|
114
|
+
| score | number | no | Response score (0-1) |
|
|
115
|
+
| difficulty | number | no | Item difficulty (0-1) |
|
|
116
|
+
| epistemicMode | EpistemicMode | no | How student arrived at answer |
|
|
117
|
+
| ext | Record | no | Extension namespace |
|
|
118
|
+
|
|
119
|
+
### Enums
|
|
120
|
+
|
|
121
|
+
**ScaffoldType:** `none`, `socratic`, `probing`, `metacognitive`, `scaffolding`, `explain`, `hint`, `demonstrate`
|
|
122
|
+
|
|
123
|
+
**EpistemicMode:** `experience`, `inference`, `analogy`, `testimony`
|
|
124
|
+
|
|
125
|
+
**DepthLevel:** `surface`, `conceptual`, `transfer`, `integration`
|
|
126
|
+
|
|
127
|
+
**TriageResult:** `correct`, `slip`, `misconception`, `disengagement`, `ambiguous`
|
|
128
|
+
|
|
129
|
+
## Verifier
|
|
130
|
+
|
|
131
|
+
The verifier checks:
|
|
132
|
+
- Schema validity (required fields, correct types)
|
|
133
|
+
- Strictly monotonic version numbers
|
|
134
|
+
- Timestamp monotonicity (configurable tolerance)
|
|
135
|
+
- Hash-chain integrity (prevHash linkage, entryHash recompute)
|
|
136
|
+
- Posterior chain consistency (each prior = previous updated)
|
|
137
|
+
- Compaction-boundary integrity
|
|
138
|
+
- Enum validity
|
|
139
|
+
|
|
140
|
+
What it deliberately **cannot** check:
|
|
141
|
+
- Whether posteriors are correctly computed (that requires the engine)
|
|
142
|
+
- Whether evidential weights are appropriate (those are calibrated server-side)
|
|
143
|
+
- Whether triage classifications are accurate (those use rules not in this package)
|
|
144
|
+
|
|
145
|
+
This is the design: **verify instead of trust**.
|
|
146
|
+
|
|
147
|
+
## Privacy
|
|
148
|
+
|
|
149
|
+
- `studentId` MUST be a pseudonymous identifier. Never include names, emails, or other PII.
|
|
150
|
+
- Events are buffered locally and sent via sendBeacon/fetch. No cookies are set.
|
|
151
|
+
- The SDK does not store any data persistently.
|
|
152
|
+
|
|
153
|
+
## Versioning
|
|
154
|
+
|
|
155
|
+
- Semver from 0.1.0.
|
|
156
|
+
- Schema changes in minor versions (0.2, 0.3).
|
|
157
|
+
- Breaking changes in major versions (1.0).
|
|
158
|
+
- Pin your dependency version.
|
|
159
|
+
|
|
160
|
+
## Links
|
|
161
|
+
|
|
162
|
+
- [Developer portal](https://play.quantumlearningmachines.com/developer)
|
|
163
|
+
- [Open Measurement Manifesto](https://play.quantumlearningmachines.com/resources/open-measurement-layer)
|
|
164
|
+
- [Misconception ontology](https://play.quantumlearningmachines.com/developer)
|
|
165
|
+
- [Model cards on Hugging Face](https://huggingface.co/QuantumLearningMachines)
|
|
166
|
+
|
|
167
|
+
## License
|
|
168
|
+
|
|
169
|
+
Apache-2.0. See [LICENSE](./LICENSE).
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
README.md
|
|
2
|
+
pyproject.toml
|
|
3
|
+
qlm_measure/__init__.py
|
|
4
|
+
qlm_measure/clients.py
|
|
5
|
+
qlm_measure/schema.py
|
|
6
|
+
qlm_measure/verifier.py
|
|
7
|
+
qlm_measure.egg-info/PKG-INFO
|
|
8
|
+
qlm_measure.egg-info/SOURCES.txt
|
|
9
|
+
qlm_measure.egg-info/dependency_links.txt
|
|
10
|
+
qlm_measure.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
qlm_measure
|