@semiont/make-meaning 0.2.30-build.59 → 0.2.30-build.61
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +88 -583
- package/dist/index.d.ts +175 -7
- package/dist/index.js +1334 -54
- package/dist/index.js.map +1 -1
- package/package.json +3 -2
package/README.md
CHANGED
|
@@ -8,42 +8,26 @@
|
|
|
8
8
|
|
|
9
9
|
**Making meaning from resources through context assembly, pattern detection, and relationship reasoning.**
|
|
10
10
|
|
|
11
|
-
This package transforms raw resources into meaningful, interconnected knowledge
|
|
11
|
+
This package transforms raw resources into meaningful, interconnected knowledge through:
|
|
12
12
|
|
|
13
13
|
- **Context Assembly**: Gathering resource metadata, content, and annotations from distributed storage
|
|
14
14
|
- **Pattern Detection**: AI-powered discovery of semantic patterns (comments, highlights, assessments, tags)
|
|
15
|
-
- **
|
|
15
|
+
- **Relationship Reasoning**: Navigating connections between resources through graph traversal
|
|
16
|
+
- **Job Workers**: Asynchronous processing of detection tasks with progress tracking
|
|
16
17
|
|
|
17
|
-
##
|
|
18
|
-
|
|
19
|
-
Resources don't exist in isolation. A document becomes meaningful when we understand its annotations, its relationships to other resources, and the patterns within its content. `@semiont/make-meaning` provides the infrastructure to assemble this context from event-sourced storage, detect semantic patterns using AI, and reason about resource relationships through graph traversal.
|
|
20
|
-
|
|
21
|
-
This is the "applied meaning-making" layer - it sits between low-level AI primitives ([@semiont/inference](../inference/)) and high-level application orchestration ([apps/backend](../../apps/backend/)).
|
|
22
|
-
|
|
23
|
-
## Installation
|
|
18
|
+
## Quick Start
|
|
24
19
|
|
|
25
20
|
```bash
|
|
26
21
|
npm install @semiont/make-meaning
|
|
27
22
|
```
|
|
28
23
|
|
|
29
|
-
## Quick Start
|
|
30
|
-
|
|
31
24
|
### Assemble Resource Context
|
|
32
25
|
|
|
33
26
|
```typescript
|
|
34
27
|
import { ResourceContext } from '@semiont/make-meaning';
|
|
35
|
-
import type { EnvironmentConfig } from '@semiont/core';
|
|
36
28
|
|
|
37
|
-
// Get resource metadata from event-sourced view storage
|
|
38
29
|
const resource = await ResourceContext.getResourceMetadata(resourceId, config);
|
|
39
|
-
|
|
40
|
-
// List all resources with optional filtering
|
|
41
|
-
const resources = await ResourceContext.listResources(
|
|
42
|
-
{ createdAfter: '2024-01-01' },
|
|
43
|
-
config
|
|
44
|
-
);
|
|
45
|
-
|
|
46
|
-
// Add content previews to resource descriptors
|
|
30
|
+
const resources = await ResourceContext.listResources({ createdAfter: '2024-01-01' }, config);
|
|
47
31
|
const withContent = await ResourceContext.addContentPreviews(resources, config);
|
|
48
32
|
```
|
|
49
33
|
|
|
@@ -62,13 +46,6 @@ const context = await AnnotationContext.buildLLMContext(
|
|
|
62
46
|
config,
|
|
63
47
|
{ contextLines: 5 }
|
|
64
48
|
);
|
|
65
|
-
|
|
66
|
-
// Generate AI summary of an annotation
|
|
67
|
-
const summary = await AnnotationContext.generateAnnotationSummary(
|
|
68
|
-
annotationId,
|
|
69
|
-
resourceId,
|
|
70
|
-
config
|
|
71
|
-
);
|
|
72
49
|
```
|
|
73
50
|
|
|
74
51
|
### Detect Semantic Patterns
|
|
@@ -93,15 +70,6 @@ const highlights = await AnnotationDetection.detectHighlights(
|
|
|
93
70
|
0.5
|
|
94
71
|
);
|
|
95
72
|
|
|
96
|
-
// Detect passages that merit assessment/evaluation
|
|
97
|
-
const assessments = await AnnotationDetection.detectAssessments(
|
|
98
|
-
resourceId,
|
|
99
|
-
config,
|
|
100
|
-
'Evaluate clarity and technical accuracy',
|
|
101
|
-
'constructive',
|
|
102
|
-
0.6
|
|
103
|
-
);
|
|
104
|
-
|
|
105
73
|
// Detect and extract structured tags from text using ontology schemas
|
|
106
74
|
const tags = await AnnotationDetection.detectTags(
|
|
107
75
|
resourceId,
|
|
@@ -111,41 +79,6 @@ const tags = await AnnotationDetection.detectTags(
|
|
|
111
79
|
);
|
|
112
80
|
```
|
|
113
81
|
|
|
114
|
-
### Structured Tagging with Ontology Schemas
|
|
115
|
-
|
|
116
|
-
A powerful use case is **structured tagging** using tag schemas defined in [@semiont/ontology](../ontology/). For example, legal writing can be analyzed using the IRAC framework (Issue, Rule, Application, Conclusion):
|
|
117
|
-
|
|
118
|
-
```typescript
|
|
119
|
-
import { AnnotationDetection } from '@semiont/make-meaning';
|
|
120
|
-
|
|
121
|
-
// Analyze a legal brief using IRAC schema
|
|
122
|
-
const categories = ['issue', 'rule', 'application', 'conclusion'];
|
|
123
|
-
|
|
124
|
-
for (const category of categories) {
|
|
125
|
-
const tags = await AnnotationDetection.detectTags(
|
|
126
|
-
resourceId,
|
|
127
|
-
config,
|
|
128
|
-
'irac', // Tag schema from @semiont/ontology
|
|
129
|
-
category // Which category to detect
|
|
130
|
-
);
|
|
131
|
-
|
|
132
|
-
console.log(`Found ${tags.length} ${category} passages`);
|
|
133
|
-
}
|
|
134
|
-
```
|
|
135
|
-
|
|
136
|
-
**Why this matters:**
|
|
137
|
-
|
|
138
|
-
When you tag multiple documents with the same schema (e.g., IRAC for legal briefs, IMRAD for scientific papers), you create a **structured semantic layer** across your corpus:
|
|
139
|
-
|
|
140
|
-
- **Rich traversal**: Find all "issue" statements across 100 legal briefs
|
|
141
|
-
- **Cross-document analysis**: Compare how different authors structure their "application" sections
|
|
142
|
-
- **Context retrieval**: When reading one brief, see related "rule" passages from other cases
|
|
143
|
-
- **Graph-based reasoning**: Trace argument patterns across your entire document collection
|
|
144
|
-
|
|
145
|
-
This transforms a collection of unstructured documents into a queryable knowledge base organized by domain-specific rhetorical structures.
|
|
146
|
-
|
|
147
|
-
See [@semiont/ontology](../ontology/) for available tag schemas and how to define custom schemas.
|
|
148
|
-
|
|
149
82
|
### Navigate Resource Relationships
|
|
150
83
|
|
|
151
84
|
```typescript
|
|
@@ -157,473 +90,124 @@ const backlinks = await GraphContext.getBacklinks(resourceId, config);
|
|
|
157
90
|
// Find shortest path between two resources
|
|
158
91
|
const paths = await GraphContext.findPath(fromResourceId, toResourceId, config, 3);
|
|
159
92
|
|
|
160
|
-
// Get all connections for a resource
|
|
161
|
-
const connections = await GraphContext.getResourceConnections(resourceId, config);
|
|
162
|
-
|
|
163
93
|
// Full-text search across all resources
|
|
164
94
|
const results = await GraphContext.searchResources('neural networks', config, 10);
|
|
165
95
|
```
|
|
166
96
|
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
`@semiont/make-meaning` implements a **three-layer architecture**:
|
|
97
|
+
### Use Job Workers
|
|
170
98
|
|
|
171
|
-
```
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
│ @semiont/make-meaning │
|
|
180
|
-
│ Context assembly, pattern detection, │
|
|
181
|
-
│ relationship reasoning │
|
|
182
|
-
└─────────────────────────────────────────────┘
|
|
183
|
-
↓
|
|
184
|
-
┌─────────────────────────────────────────────┐
|
|
185
|
-
│ @semiont/inference │
|
|
186
|
-
│ AI primitives: prompts, parsers, │
|
|
187
|
-
│ generateText abstraction │
|
|
188
|
-
└─────────────────────────────────────────────┘
|
|
189
|
-
```
|
|
190
|
-
|
|
191
|
-
**Key principles:**
|
|
192
|
-
|
|
193
|
-
- **Event-sourced context**: Resources and annotations are assembled from event streams via view storage
|
|
194
|
-
- **Content-addressed storage**: Content retrieved using checksums, enabling deduplication and caching
|
|
195
|
-
- **Graph-backed relationships**: @semiont/graph provides graph traversal for backlinks, paths, and connections
|
|
196
|
-
- **Separation of concerns**: Detection logic (make-meaning) is separate from job orchestration (backend)
|
|
197
|
-
|
|
198
|
-
See [MAKE-MEANING-PACKAGE.md](../../MAKE-MEANING-PACKAGE.md) for complete architecture documentation.
|
|
199
|
-
|
|
200
|
-
## API Reference
|
|
99
|
+
```typescript
|
|
100
|
+
import {
|
|
101
|
+
ReferenceDetectionWorker,
|
|
102
|
+
HighlightDetectionWorker,
|
|
103
|
+
GenerationWorker,
|
|
104
|
+
} from '@semiont/make-meaning';
|
|
105
|
+
import { JobQueue } from '@semiont/jobs';
|
|
106
|
+
import { createEventStore } from '@semiont/event-sourcing';
|
|
201
107
|
|
|
202
|
-
|
|
108
|
+
// Create shared dependencies
|
|
109
|
+
const jobQueue = new JobQueue({ dataDir: './data' });
|
|
110
|
+
await jobQueue.initialize();
|
|
111
|
+
const eventStore = await createEventStore(config);
|
|
203
112
|
|
|
204
|
-
|
|
113
|
+
// Create workers with explicit dependencies
|
|
114
|
+
const referenceWorker = new ReferenceDetectionWorker(jobQueue, config, eventStore);
|
|
115
|
+
const highlightWorker = new HighlightDetectionWorker(jobQueue, config, eventStore);
|
|
116
|
+
const generationWorker = new GenerationWorker(jobQueue, config, eventStore);
|
|
205
117
|
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
static async getResourceMetadata(
|
|
213
|
-
resourceId: ResourceId,
|
|
214
|
-
config: EnvironmentConfig
|
|
215
|
-
): Promise<ResourceDescriptor | null>
|
|
216
|
-
|
|
217
|
-
/**
|
|
218
|
-
* List resources with optional filtering
|
|
219
|
-
* Implementation: packages/make-meaning/src/resource-context.ts:30-48
|
|
220
|
-
*/
|
|
221
|
-
static async listResources(
|
|
222
|
-
filters: ListResourcesFilters | undefined,
|
|
223
|
-
config: EnvironmentConfig
|
|
224
|
-
): Promise<ResourceDescriptor[]>
|
|
225
|
-
|
|
226
|
-
/**
|
|
227
|
-
* Add content previews to resource descriptors
|
|
228
|
-
* Implementation: packages/make-meaning/src/resource-context.ts:50-77
|
|
229
|
-
*/
|
|
230
|
-
static async addContentPreviews(
|
|
231
|
-
resources: ResourceDescriptor[],
|
|
232
|
-
config: EnvironmentConfig
|
|
233
|
-
): Promise<Array<ResourceDescriptor & { content: string }>>
|
|
234
|
-
}
|
|
118
|
+
// Start workers
|
|
119
|
+
await Promise.all([
|
|
120
|
+
referenceWorker.start(),
|
|
121
|
+
highlightWorker.start(),
|
|
122
|
+
generationWorker.start(),
|
|
123
|
+
]);
|
|
235
124
|
```
|
|
236
125
|
|
|
237
|
-
|
|
238
|
-
```typescript
|
|
239
|
-
interface ListResourcesFilters {
|
|
240
|
-
createdAfter?: string;
|
|
241
|
-
createdBefore?: string;
|
|
242
|
-
mimeType?: string;
|
|
243
|
-
limit?: number;
|
|
244
|
-
}
|
|
245
|
-
```
|
|
126
|
+
## Documentation
|
|
246
127
|
|
|
247
|
-
|
|
128
|
+
- **[API Reference](./docs/api-reference.md)** - Complete API documentation for all classes and methods
|
|
129
|
+
- **[Job Workers](./docs/job-workers.md)** - Asynchronous task processing with progress tracking
|
|
130
|
+
- **[Architecture](./docs/architecture.md)** - System design and data flow
|
|
131
|
+
- **[Examples](./docs/examples.md)** - Common use cases and patterns
|
|
248
132
|
|
|
249
|
-
|
|
133
|
+
## Philosophy
|
|
250
134
|
|
|
251
|
-
|
|
252
|
-
class AnnotationContext {
|
|
253
|
-
/**
|
|
254
|
-
* Build LLM context for an annotation (includes surrounding text)
|
|
255
|
-
* Implementation: packages/make-meaning/src/annotation-context.ts:35-120
|
|
256
|
-
*/
|
|
257
|
-
static async buildLLMContext(
|
|
258
|
-
annotationUri: AnnotationUri,
|
|
259
|
-
resourceId: ResourceId,
|
|
260
|
-
config: EnvironmentConfig,
|
|
261
|
-
options: BuildContextOptions
|
|
262
|
-
): Promise<AnnotationLLMContextResponse>
|
|
263
|
-
|
|
264
|
-
/**
|
|
265
|
-
* Get all annotations for a resource, organized by motivation
|
|
266
|
-
* Implementation: packages/make-meaning/src/annotation-context.ts:122-172
|
|
267
|
-
*/
|
|
268
|
-
static async getResourceAnnotations(
|
|
269
|
-
resourceId: ResourceId,
|
|
270
|
-
config: EnvironmentConfig
|
|
271
|
-
): Promise<ResourceAnnotations>
|
|
272
|
-
|
|
273
|
-
/**
|
|
274
|
-
* Get all annotations for a resource (flat list)
|
|
275
|
-
* Implementation: packages/make-meaning/src/annotation-context.ts:174-187
|
|
276
|
-
*/
|
|
277
|
-
static async getAllAnnotations(
|
|
278
|
-
resourceId: ResourceId,
|
|
279
|
-
config: EnvironmentConfig
|
|
280
|
-
): Promise<Annotation[]>
|
|
281
|
-
|
|
282
|
-
/**
|
|
283
|
-
* Get a specific annotation by ID
|
|
284
|
-
* Implementation: packages/make-meaning/src/annotation-context.ts:189-202
|
|
285
|
-
*/
|
|
286
|
-
static async getAnnotation(
|
|
287
|
-
annotationId: AnnotationId,
|
|
288
|
-
resourceId: ResourceId,
|
|
289
|
-
config: EnvironmentConfig
|
|
290
|
-
): Promise<Annotation | null>
|
|
291
|
-
|
|
292
|
-
/**
|
|
293
|
-
* List annotations with optional filtering
|
|
294
|
-
* Implementation: packages/make-meaning/src/annotation-context.ts:204-225
|
|
295
|
-
*/
|
|
296
|
-
static async listAnnotations(
|
|
297
|
-
filters: { resourceId?: ResourceId; type?: AnnotationCategory } | undefined,
|
|
298
|
-
config: EnvironmentConfig
|
|
299
|
-
): Promise<Annotation[]>
|
|
300
|
-
|
|
301
|
-
/**
|
|
302
|
-
* Check if a resource exists in view storage
|
|
303
|
-
* Implementation: packages/make-meaning/src/annotation-context.ts:227-236
|
|
304
|
-
*/
|
|
305
|
-
static async resourceExists(
|
|
306
|
-
resourceId: ResourceId,
|
|
307
|
-
config: EnvironmentConfig
|
|
308
|
-
): Promise<boolean>
|
|
309
|
-
|
|
310
|
-
/**
|
|
311
|
-
* Get resource statistics (version, last updated)
|
|
312
|
-
* Implementation: packages/make-meaning/src/annotation-context.ts:238-254
|
|
313
|
-
*/
|
|
314
|
-
static async getResourceStats(
|
|
315
|
-
resourceId: ResourceId,
|
|
316
|
-
config: EnvironmentConfig
|
|
317
|
-
): Promise<{
|
|
318
|
-
resourceId: ResourceId;
|
|
319
|
-
version: number;
|
|
320
|
-
updatedAt: string;
|
|
321
|
-
}>
|
|
322
|
-
|
|
323
|
-
/**
|
|
324
|
-
* Get annotation context (surrounding text)
|
|
325
|
-
* Implementation: packages/make-meaning/src/annotation-context.ts:256-314
|
|
326
|
-
*/
|
|
327
|
-
static async getAnnotationContext(
|
|
328
|
-
annotationId: AnnotationId,
|
|
329
|
-
resourceId: ResourceId,
|
|
330
|
-
contextBefore: number,
|
|
331
|
-
contextAfter: number,
|
|
332
|
-
config: EnvironmentConfig
|
|
333
|
-
): Promise<AnnotationContextResponse>
|
|
334
|
-
|
|
335
|
-
/**
|
|
336
|
-
* Generate AI summary of an annotation
|
|
337
|
-
* Implementation: packages/make-meaning/src/annotation-context.ts:316-381
|
|
338
|
-
*/
|
|
339
|
-
static async generateAnnotationSummary(
|
|
340
|
-
annotationId: AnnotationId,
|
|
341
|
-
resourceId: ResourceId,
|
|
342
|
-
config: EnvironmentConfig
|
|
343
|
-
): Promise<ContextualSummaryResponse>
|
|
344
|
-
}
|
|
345
|
-
```
|
|
135
|
+
Resources don't exist in isolation. A document becomes meaningful when we understand its annotations, its relationships to other resources, and the patterns within its content. `@semiont/make-meaning` provides the infrastructure to:
|
|
346
136
|
|
|
347
|
-
**
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
contextLines?: number; // Lines of surrounding text (default: 5)
|
|
351
|
-
includeMetadata?: boolean; // Include resource metadata (default: true)
|
|
352
|
-
}
|
|
353
|
-
```
|
|
137
|
+
1. **Assemble context** from event-sourced storage
|
|
138
|
+
2. **Detect patterns** using AI inference
|
|
139
|
+
3. **Reason about relationships** through graph traversal
|
|
354
140
|
|
|
355
|
-
|
|
141
|
+
This is the "applied meaning-making" layer - it sits between low-level AI primitives ([@semiont/inference](../inference/)) and high-level application orchestration ([apps/backend](../../apps/backend/)).
|
|
356
142
|
|
|
357
|
-
|
|
143
|
+
## Architecture
|
|
358
144
|
|
|
359
|
-
|
|
360
|
-
class GraphContext {
|
|
361
|
-
/**
|
|
362
|
-
* Get all resources referencing this resource (backlinks)
|
|
363
|
-
* Requires graph traversal - uses @semiont/graph
|
|
364
|
-
* Implementation: packages/make-meaning/src/graph-context.ts:26-30
|
|
365
|
-
*/
|
|
366
|
-
static async getBacklinks(
|
|
367
|
-
resourceId: ResourceId,
|
|
368
|
-
config: EnvironmentConfig
|
|
369
|
-
): Promise<Annotation[]>
|
|
370
|
-
|
|
371
|
-
/**
|
|
372
|
-
* Find shortest path between two resources
|
|
373
|
-
* Requires graph traversal - uses @semiont/graph
|
|
374
|
-
* Implementation: packages/make-meaning/src/graph-context.ts:36-44
|
|
375
|
-
*/
|
|
376
|
-
static async findPath(
|
|
377
|
-
fromResourceId: ResourceId,
|
|
378
|
-
toResourceId: ResourceId,
|
|
379
|
-
config: EnvironmentConfig,
|
|
380
|
-
maxDepth?: number
|
|
381
|
-
): Promise<GraphPath[]>
|
|
382
|
-
|
|
383
|
-
/**
|
|
384
|
-
* Get resource connections (graph edges)
|
|
385
|
-
* Requires graph traversal - uses @semiont/graph
|
|
386
|
-
* Implementation: packages/make-meaning/src/graph-context.ts:50-53
|
|
387
|
-
*/
|
|
388
|
-
static async getResourceConnections(
|
|
389
|
-
resourceId: ResourceId,
|
|
390
|
-
config: EnvironmentConfig
|
|
391
|
-
): Promise<GraphConnection[]>
|
|
392
|
-
|
|
393
|
-
/**
|
|
394
|
-
* Search resources by name (cross-resource query)
|
|
395
|
-
* Requires full-text search - uses @semiont/graph
|
|
396
|
-
* Implementation: packages/make-meaning/src/graph-context.ts:59-62
|
|
397
|
-
*/
|
|
398
|
-
static async searchResources(
|
|
399
|
-
query: string,
|
|
400
|
-
config: EnvironmentConfig,
|
|
401
|
-
limit?: number
|
|
402
|
-
): Promise<ResourceDescriptor[]>
|
|
403
|
-
}
|
|
404
|
-
```
|
|
145
|
+
Three-layer design separating concerns:
|
|
405
146
|
|
|
406
|
-
|
|
147
|
+
```mermaid
|
|
148
|
+
graph TB
|
|
149
|
+
Backend["<b>apps/backend</b><br/>Job orchestration, HTTP APIs, streaming"]
|
|
150
|
+
MakeMeaning["<b>@semiont/make-meaning</b><br/>Context assembly, pattern detection,<br/>relationship reasoning, job workers"]
|
|
151
|
+
Inference["<b>@semiont/inference</b><br/>AI primitives: prompts, parsers,<br/>generateText abstraction"]
|
|
407
152
|
|
|
408
|
-
|
|
153
|
+
Backend --> MakeMeaning
|
|
154
|
+
MakeMeaning --> Inference
|
|
409
155
|
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
* Detect passages that merit commentary
|
|
414
|
-
* Implementation: packages/make-meaning/src/annotation-detection.ts:27-65
|
|
415
|
-
* Uses: MotivationPrompts.buildCommentPrompt, MotivationParsers.parseComments
|
|
416
|
-
*/
|
|
417
|
-
static async detectComments(
|
|
418
|
-
resourceId: ResourceId,
|
|
419
|
-
config: EnvironmentConfig,
|
|
420
|
-
instructions?: string,
|
|
421
|
-
tone?: string,
|
|
422
|
-
density?: number
|
|
423
|
-
): Promise<CommentMatch[]>
|
|
424
|
-
|
|
425
|
-
/**
|
|
426
|
-
* Detect passages that should be highlighted
|
|
427
|
-
* Implementation: packages/make-meaning/src/annotation-detection.ts:67-101
|
|
428
|
-
* Uses: MotivationPrompts.buildHighlightPrompt, MotivationParsers.parseHighlights
|
|
429
|
-
*/
|
|
430
|
-
static async detectHighlights(
|
|
431
|
-
resourceId: ResourceId,
|
|
432
|
-
config: EnvironmentConfig,
|
|
433
|
-
instructions?: string,
|
|
434
|
-
density?: number
|
|
435
|
-
): Promise<HighlightMatch[]>
|
|
436
|
-
|
|
437
|
-
/**
|
|
438
|
-
* Detect passages that merit assessment/evaluation
|
|
439
|
-
* Implementation: packages/make-meaning/src/annotation-detection.ts:103-141
|
|
440
|
-
* Uses: MotivationPrompts.buildAssessmentPrompt, MotivationParsers.parseAssessments
|
|
441
|
-
*/
|
|
442
|
-
static async detectAssessments(
|
|
443
|
-
resourceId: ResourceId,
|
|
444
|
-
config: EnvironmentConfig,
|
|
445
|
-
instructions?: string,
|
|
446
|
-
tone?: string,
|
|
447
|
-
density?: number
|
|
448
|
-
): Promise<AssessmentMatch[]>
|
|
449
|
-
|
|
450
|
-
/**
|
|
451
|
-
* Detect and extract structured tags from text
|
|
452
|
-
* Implementation: packages/make-meaning/src/annotation-detection.ts:143-197
|
|
453
|
-
* Uses: MotivationPrompts.buildTagPrompt, MotivationParsers.parseTags
|
|
454
|
-
*/
|
|
455
|
-
static async detectTags(
|
|
456
|
-
resourceId: ResourceId,
|
|
457
|
-
config: EnvironmentConfig,
|
|
458
|
-
schemaId: string,
|
|
459
|
-
category: string
|
|
460
|
-
): Promise<TagMatch[]>
|
|
461
|
-
}
|
|
156
|
+
style Backend fill:#e1f5ff
|
|
157
|
+
style MakeMeaning fill:#fff4e6
|
|
158
|
+
style Inference fill:#f3e5f5
|
|
462
159
|
```
|
|
463
160
|
|
|
464
|
-
**
|
|
465
|
-
```typescript
|
|
466
|
-
// Re-exported from @semiont/inference for convenience
|
|
467
|
-
interface CommentMatch {
|
|
468
|
-
exact: string; // The exact text passage
|
|
469
|
-
start: number; // Character offset start
|
|
470
|
-
end: number; // Character offset end
|
|
471
|
-
prefix?: string; // Context before (for fuzzy anchoring)
|
|
472
|
-
suffix?: string; // Context after (for fuzzy anchoring)
|
|
473
|
-
comment: string; // The AI-generated comment
|
|
474
|
-
}
|
|
475
|
-
|
|
476
|
-
interface HighlightMatch {
|
|
477
|
-
exact: string;
|
|
478
|
-
start: number;
|
|
479
|
-
end: number;
|
|
480
|
-
prefix?: string;
|
|
481
|
-
suffix?: string;
|
|
482
|
-
}
|
|
483
|
-
|
|
484
|
-
interface AssessmentMatch {
|
|
485
|
-
exact: string;
|
|
486
|
-
start: number;
|
|
487
|
-
end: number;
|
|
488
|
-
prefix?: string;
|
|
489
|
-
suffix?: string;
|
|
490
|
-
assessment: string; // The AI-generated assessment
|
|
491
|
-
}
|
|
492
|
-
|
|
493
|
-
interface TagMatch {
|
|
494
|
-
exact: string;
|
|
495
|
-
start: number;
|
|
496
|
-
end: number;
|
|
497
|
-
prefix?: string;
|
|
498
|
-
suffix?: string;
|
|
499
|
-
category: string; // The tag category
|
|
500
|
-
}
|
|
501
|
-
```
|
|
161
|
+
**Key principles:**
|
|
502
162
|
|
|
503
|
-
**
|
|
163
|
+
- **Event-sourced context**: Resources and annotations assembled from event streams
|
|
164
|
+
- **Content-addressed storage**: Content retrieved using checksums (deduplication, caching)
|
|
165
|
+
- **Graph-backed relationships**: @semiont/graph provides traversal for backlinks, paths, connections
|
|
166
|
+
- **Explicit dependencies**: Workers receive JobQueue and EventStore via constructor (no singletons)
|
|
504
167
|
|
|
505
|
-
|
|
506
|
-
- `tone`: Tone for comments/assessments (e.g., "educational", "constructive", "analytical")
|
|
507
|
-
- `density`: Target density 0.0-1.0 (0.5 = ~50% of passages should be detected)
|
|
168
|
+
See [Architecture](./docs/architecture.md) for complete details.
|
|
508
169
|
|
|
509
|
-
##
|
|
170
|
+
## Exports
|
|
510
171
|
|
|
511
|
-
###
|
|
172
|
+
### Context Assembly
|
|
512
173
|
|
|
513
|
-
|
|
514
|
-
|
|
174
|
+
- `ResourceContext` - Resource metadata and content
|
|
175
|
+
- `AnnotationContext` - Annotation queries and context building
|
|
176
|
+
- `GraphContext` - Graph traversal and search
|
|
515
177
|
|
|
516
|
-
|
|
517
|
-
const context = await AnnotationContext.buildLLMContext(
|
|
518
|
-
annotationUri,
|
|
519
|
-
resourceId,
|
|
520
|
-
config,
|
|
521
|
-
{ contextLines: 10 }
|
|
522
|
-
);
|
|
178
|
+
### Pattern Detection
|
|
523
179
|
|
|
524
|
-
|
|
525
|
-
// - The annotation itself
|
|
526
|
-
// - Surrounding text (10 lines before/after)
|
|
527
|
-
// - Resource metadata
|
|
528
|
-
// - Related annotations in the vicinity
|
|
529
|
-
```
|
|
180
|
+
- `AnnotationDetection` - AI-powered semantic pattern detection
|
|
530
181
|
|
|
531
|
-
###
|
|
182
|
+
### Job Workers
|
|
532
183
|
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
184
|
+
- `ReferenceDetectionWorker` - Entity reference detection
|
|
185
|
+
- `GenerationWorker` - AI content generation
|
|
186
|
+
- `HighlightDetectionWorker` - Highlight detection
|
|
187
|
+
- `CommentDetectionWorker` - Comment detection
|
|
188
|
+
- `AssessmentDetectionWorker` - Assessment detection
|
|
189
|
+
- `TagDetectionWorker` - Structured tag detection
|
|
536
190
|
|
|
537
|
-
|
|
538
|
-
const highlights = await AnnotationDetection.detectHighlights(
|
|
539
|
-
resourceId,
|
|
540
|
-
config,
|
|
541
|
-
'Find key definitions and important claims',
|
|
542
|
-
0.6 // Medium density
|
|
543
|
-
);
|
|
191
|
+
See [Job Workers](./docs/job-workers.md) for implementation details.
|
|
544
192
|
|
|
545
|
-
|
|
546
|
-
const eventStore = await createEventStore(config);
|
|
547
|
-
for (const highlight of highlights) {
|
|
548
|
-
const annotation = {
|
|
549
|
-
'@context': 'http://www.w3.org/ns/anno.jsonld',
|
|
550
|
-
type: 'Annotation',
|
|
551
|
-
id: generateAnnotationId(),
|
|
552
|
-
motivation: 'highlighting',
|
|
553
|
-
target: {
|
|
554
|
-
type: 'SpecificResource',
|
|
555
|
-
source: resourceUri,
|
|
556
|
-
selector: [
|
|
557
|
-
{
|
|
558
|
-
type: 'TextPositionSelector',
|
|
559
|
-
start: highlight.start,
|
|
560
|
-
end: highlight.end,
|
|
561
|
-
},
|
|
562
|
-
{
|
|
563
|
-
type: 'TextQuoteSelector',
|
|
564
|
-
exact: highlight.exact,
|
|
565
|
-
prefix: highlight.prefix,
|
|
566
|
-
suffix: highlight.suffix,
|
|
567
|
-
},
|
|
568
|
-
],
|
|
569
|
-
},
|
|
570
|
-
body: [],
|
|
571
|
-
};
|
|
572
|
-
|
|
573
|
-
await eventStore.appendEvent({
|
|
574
|
-
type: 'annotation.added',
|
|
575
|
-
resourceId,
|
|
576
|
-
userId,
|
|
577
|
-
version: 1,
|
|
578
|
-
payload: { annotation },
|
|
579
|
-
});
|
|
580
|
-
}
|
|
581
|
-
```
|
|
582
|
-
|
|
583
|
-
### Navigating Resource Relationships
|
|
193
|
+
### Type Re-exports
|
|
584
194
|
|
|
585
195
|
```typescript
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
// Find connection path between two resources
|
|
593
|
-
const paths = await GraphContext.findPath(sourceId, targetId, config, 3);
|
|
594
|
-
if (paths.length > 0) {
|
|
595
|
-
console.log(`Shortest path has ${paths[0].nodes.length} nodes`);
|
|
596
|
-
}
|
|
597
|
-
|
|
598
|
-
// Get all connections for a resource
|
|
599
|
-
const connections = await GraphContext.getResourceConnections(resourceId, config);
|
|
600
|
-
// connections = [{ from: ResourceId, to: ResourceId, via: AnnotationId }, ...]
|
|
196
|
+
export type {
|
|
197
|
+
CommentMatch,
|
|
198
|
+
HighlightMatch,
|
|
199
|
+
AssessmentMatch,
|
|
200
|
+
TagMatch,
|
|
201
|
+
} from '@semiont/inference';
|
|
601
202
|
```
|
|
602
203
|
|
|
603
204
|
## Configuration
|
|
604
205
|
|
|
605
|
-
All methods require an `EnvironmentConfig` object
|
|
206
|
+
All methods require an `EnvironmentConfig` object:
|
|
606
207
|
|
|
607
208
|
```typescript
|
|
608
|
-
|
|
609
|
-
services: {
|
|
610
|
-
backend: {
|
|
611
|
-
publicURL: string; // Base URL for resource URIs
|
|
612
|
-
};
|
|
613
|
-
openai?: {
|
|
614
|
-
apiKey: string; // Required for detection methods
|
|
615
|
-
model?: string; // Default: 'gpt-4o-mini'
|
|
616
|
-
temperature?: number; // Default: 0.7
|
|
617
|
-
};
|
|
618
|
-
};
|
|
619
|
-
storage: {
|
|
620
|
-
base: string; // Base path for filesystem storage
|
|
621
|
-
};
|
|
622
|
-
}
|
|
623
|
-
```
|
|
209
|
+
import type { EnvironmentConfig } from '@semiont/core';
|
|
624
210
|
|
|
625
|
-
Example:
|
|
626
|
-
```typescript
|
|
627
211
|
const config: EnvironmentConfig = {
|
|
628
212
|
services: {
|
|
629
213
|
backend: {
|
|
@@ -652,95 +236,16 @@ const config: EnvironmentConfig = {
|
|
|
652
236
|
- **[@semiont/graph](../graph/)**: Neo4j graph database client
|
|
653
237
|
- **[@semiont/ontology](../ontology/)**: Schema definitions for tags
|
|
654
238
|
- **[@semiont/inference](../inference/)**: AI primitives (prompts, parsers, generateText)
|
|
239
|
+
- **[@semiont/jobs](../jobs/)**: Job queue and worker base class
|
|
655
240
|
|
|
656
|
-
##
|
|
657
|
-
|
|
658
|
-
See [packages/inference/README.md](../inference/README.md) for details on the AI pipeline.
|
|
659
|
-
|
|
660
|
-
**High-level flow:**
|
|
661
|
-
|
|
662
|
-
1. **Context Assembly**: `ResourceContext.getResourceMetadata()` retrieves resource content
|
|
663
|
-
2. **Prompt Building**: `MotivationPrompts.buildXPrompt()` creates AI prompt with domain knowledge
|
|
664
|
-
3. **AI Inference**: `generateText()` calls OpenAI API with prompt
|
|
665
|
-
4. **Response Parsing**: `MotivationParsers.parseX()` extracts structured matches from response
|
|
666
|
-
5. **Offset Validation**: Parser validates that `start`/`end` offsets match `exact` text in content
|
|
667
|
-
|
|
668
|
-
**Example for highlights:**
|
|
669
|
-
|
|
670
|
-
```typescript
|
|
671
|
-
// 1. Get content
|
|
672
|
-
const resource = await ResourceContext.getResourceMetadata(resourceId, config);
|
|
673
|
-
const content = await representationStore.retrieve(resource.contentId);
|
|
674
|
-
|
|
675
|
-
// 2. Build prompt
|
|
676
|
-
const prompt = MotivationPrompts.buildHighlightPrompt(
|
|
677
|
-
content,
|
|
678
|
-
'Find key definitions',
|
|
679
|
-
0.6
|
|
680
|
-
);
|
|
681
|
-
|
|
682
|
-
// 3. Generate AI response
|
|
683
|
-
const response = await generateText(prompt, config);
|
|
684
|
-
|
|
685
|
-
// 4. Parse and validate
|
|
686
|
-
const highlights = MotivationParsers.parseHighlights(response, content);
|
|
687
|
-
// Returns: HighlightMatch[] with validated offsets
|
|
688
|
-
```
|
|
689
|
-
|
|
690
|
-
## Worker Integration
|
|
241
|
+
## Testing
|
|
691
242
|
|
|
692
|
-
|
|
693
|
-
|
|
694
|
-
|
|
695
|
-
|
|
696
|
-
- [assessment-detection-worker.ts](../../apps/backend/src/jobs/workers/assessment-detection-worker.ts) - Delegated detection to `AnnotationDetection.detectAssessments()`
|
|
697
|
-
- [tag-detection-worker.ts](../../apps/backend/src/jobs/workers/tag-detection-worker.ts) - Delegated detection to `AnnotationDetection.detectTags()`
|
|
698
|
-
|
|
699
|
-
Workers handle:
|
|
700
|
-
- Job lifecycle (pending → running → completed/failed)
|
|
701
|
-
- Progress tracking and event emission
|
|
702
|
-
- Annotation creation via event store
|
|
703
|
-
- Error handling and retries
|
|
704
|
-
|
|
705
|
-
All detection logic lives in `@semiont/make-meaning`, keeping workers focused on orchestration.
|
|
706
|
-
|
|
707
|
-
## Future Direction
|
|
708
|
-
|
|
709
|
-
### Deterministic Reasoning
|
|
710
|
-
|
|
711
|
-
Future versions will add deterministic reasoning capabilities alongside AI-powered detection:
|
|
712
|
-
|
|
713
|
-
- **Rule-based pattern matching**: Detect annotations using regex, string matching, or custom predicates
|
|
714
|
-
- **Ontology-driven inference**: Apply OWL/RDFS reasoning over resource relationships
|
|
715
|
-
- **Compositional reasoning**: Combine multiple reasoning strategies (AI + rules + ontology)
|
|
716
|
-
|
|
717
|
-
Example (aspirational):
|
|
718
|
-
|
|
719
|
-
```typescript
|
|
720
|
-
// AI-powered detection
|
|
721
|
-
const aiHighlights = await AnnotationDetection.detectHighlights(resourceId, config);
|
|
722
|
-
|
|
723
|
-
// Rule-based detection
|
|
724
|
-
const ruleHighlights = await ResourceReasoning.findMatches(resourceId, {
|
|
725
|
-
pattern: /\btheorem\b.*\bproof\b/gi,
|
|
726
|
-
motivation: 'highlighting',
|
|
727
|
-
});
|
|
728
|
-
|
|
729
|
-
// Ontology-based reasoning
|
|
730
|
-
const inferences = await ResourceReasoning.inferRelationships(resourceId, {
|
|
731
|
-
ontology: 'http://example.org/math-ontology',
|
|
732
|
-
rules: ['transitive-proof-chain'],
|
|
733
|
-
});
|
|
243
|
+
```bash
|
|
244
|
+
npm test # Run tests
|
|
245
|
+
npm run test:watch # Watch mode
|
|
246
|
+
npm run test:coverage # Coverage report
|
|
734
247
|
```
|
|
735
248
|
|
|
736
|
-
This will enable hybrid approaches where AI handles semantic understanding and deterministic rules handle structural patterns.
|
|
737
|
-
|
|
738
|
-
### Enhanced Context Assembly
|
|
739
|
-
|
|
740
|
-
- **Multi-resource context**: Build context spanning multiple related resources
|
|
741
|
-
- **Temporal context**: Access historical versions of resources and annotations
|
|
742
|
-
- **Provenance tracking**: Track reasoning chains and decision paths
|
|
743
|
-
|
|
744
249
|
## License
|
|
745
250
|
|
|
746
|
-
|
|
251
|
+
Apache-2.0
|