@struktur/skill 2.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +87 -0
- package/package.json +31 -0
- package/skills/struktur/README.md +152 -0
- package/skills/struktur/SKILL.md +404 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 mateffy
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
<picture height="0">
|
|
2
|
+
<source media="(min-width: 769px)" srcset="https://raw.githubusercontent.com/mateffy/struktur/main/resources/pixel.png" width="0" height="0">
|
|
3
|
+
<img src="https://raw.githubusercontent.com/mateffy/struktur/main/resources/struktur-icon-padded.webp" alt="Struktur Logo" width="150">
|
|
4
|
+
</picture>
|
|
5
|
+
|
|
6
|
+
<div>
|
|
7
|
+
<h1>
|
|
8
|
+
<picture>
|
|
9
|
+
<source media="(max-width: 768px)" srcset="https://raw.githubusercontent.com/mateffy/struktur/main/resources/pixel.png" width="0" height="0">
|
|
10
|
+
<img src="https://raw.githubusercontent.com/mateffy/struktur/main/resources/struktur-icon-padded.webp" alt="Struktur Logo" width="225" align="left">
|
|
11
|
+
</picture>
|
|
12
|
+
Struktur Agent Skill
|
|
13
|
+
</h1>
|
|
14
|
+
<p>
|
|
15
|
+
All-in-one tool for structured data extraction using LLMs. Feed it documents, get back validated JSON. Handles parsing files, chunking, retries, merging, and deduplication — you just define the schema and choose a strategy. <br /><br />
|
|
16
|
+
<a href="https://struktur.sh/docs/quickstart" target="_blank">Quickstart</a> |
|
|
17
|
+
<a href="https://struktur.sh/docs" target="_blank">Documentation</a>
|
|
18
|
+
</p>
|
|
19
|
+
</div>
|
|
20
|
+
|
|
21
|
+
<br />
|
|
22
|
+
<br />
|
|
23
|
+
|
|
24
|
+
## @struktur/skill
|
|
25
|
+
|
|
26
|
+
Agent skill for Struktur - structured data extraction using the Vercel AI SDK. This skill teaches AI coding assistants (like Claude Code, OpenCode, Codex, etc.) how to use Struktur effectively.
|
|
27
|
+
|
|
28
|
+
## What is This?
|
|
29
|
+
|
|
30
|
+
This package contains an [Agent Skills](https://github.com/anthropics/skills) compatible skill file that works across 16+ AI agent tools. When you ask your AI agent to work with Struktur, this skill automatically loads and provides:
|
|
31
|
+
|
|
32
|
+
- **API Usage**: How to use `extract()`, build artifacts, define schemas
|
|
33
|
+
- **Strategy Selection**: When to use `simple`, `parallel`, `sequential`, `doublePass`, etc.
|
|
34
|
+
- **Schema Definition**: JSON Schema patterns and shorthand field syntax
|
|
35
|
+
- **CLI Commands**: All struktur CLI commands and options
|
|
36
|
+
- **Best Practices**: Token budgets, validation retries, merge rules
|
|
37
|
+
|
|
38
|
+
## Installation
|
|
39
|
+
|
|
40
|
+
### From npm
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
npm install @struktur/skill
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Then copy to your tool's skill directory:
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
# For Claude Code
|
|
50
|
+
cp -r node_modules/@struktur/skill/skills/struktur ~/.config/claude/skills/
|
|
51
|
+
|
|
52
|
+
# For OpenCode
|
|
53
|
+
cp -r node_modules/@struktur/skill/skills/struktur ~/.config/opencode/skills/
|
|
54
|
+
|
|
55
|
+
# For OpenAI Codex
|
|
56
|
+
cp -r node_modules/@struktur/skill/skills/struktur ~/.codex/skills/
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
### Using Skills CLI
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
npx skills add @struktur/skill
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## Supported Tools
|
|
66
|
+
|
|
67
|
+
- ✅ Claude Code
|
|
68
|
+
- ✅ OpenCode
|
|
69
|
+
- ✅ OpenAI Codex
|
|
70
|
+
- ✅ Amp
|
|
71
|
+
- ✅ VS Code (GitHub Copilot)
|
|
72
|
+
- ✅ Cursor
|
|
73
|
+
- ✅ Gemini CLI
|
|
74
|
+
- ✅ JetBrains (Junie)
|
|
75
|
+
- ✅ And 10+ more tools
|
|
76
|
+
|
|
77
|
+
## Documentation
|
|
78
|
+
|
|
79
|
+
Full documentation at **[struktur.sh](https://struktur.sh)**
|
|
80
|
+
|
|
81
|
+
- [Agent Skill Guide](https://struktur.sh/docs/skill)
|
|
82
|
+
- [Installation](https://struktur.sh/docs/skill/installation)
|
|
83
|
+
- [Usage](https://struktur.sh/docs/skill/usage)
|
|
84
|
+
|
|
85
|
+
## Repository
|
|
86
|
+
|
|
87
|
+
This package is part of the [Struktur monorepo](https://github.com/mateffy/struktur).
|
package/package.json
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@struktur/skill",
|
|
3
|
+
"version": "2.6.0",
|
|
4
|
+
"description": "Agent skill for Struktur - structured data extraction using Vercel AI SDK",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"main": "skills/struktur/SKILL.md",
|
|
7
|
+
"files": [
|
|
8
|
+
"skills/**/*"
|
|
9
|
+
],
|
|
10
|
+
"keywords": [
|
|
11
|
+
"ai",
|
|
12
|
+
"agent",
|
|
13
|
+
"skill",
|
|
14
|
+
"struktur",
|
|
15
|
+
"structured-data",
|
|
16
|
+
"extraction",
|
|
17
|
+
"llm",
|
|
18
|
+
"vercel-ai-sdk"
|
|
19
|
+
],
|
|
20
|
+
"author": "mateffy",
|
|
21
|
+
"license": "FSL-1.1-MIT",
|
|
22
|
+
"repository": {
|
|
23
|
+
"type": "git",
|
|
24
|
+
"url": "https://github.com/mateffy/struktur.git",
|
|
25
|
+
"directory": "packages/skill"
|
|
26
|
+
},
|
|
27
|
+
"bugs": {
|
|
28
|
+
"url": "https://github.com/mateffy/struktur/issues"
|
|
29
|
+
},
|
|
30
|
+
"homepage": "https://github.com/mateffy/struktur#readme"
|
|
31
|
+
}
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# Struktur Agent Skill
|
|
2
|
+
|
|
3
|
+
This directory contains the Agent Skill for Struktur - structured data extraction using the Vercel AI SDK.
|
|
4
|
+
|
|
5
|
+
## What is an Agent Skill?
|
|
6
|
+
|
|
7
|
+
An Agent Skill is a modular knowledge package that AI coding agents (like Claude Code, OpenCode, Codex, etc.) can discover and load automatically. Skills follow an open standard and work across 16+ AI agent tools.
|
|
8
|
+
|
|
9
|
+
## How It Works
|
|
10
|
+
|
|
11
|
+
When you ask your AI agent to work with Struktur, this skill automatically loads and provides:
|
|
12
|
+
|
|
13
|
+
- **API Usage**: How to use `extract()`, build artifacts, define schemas
|
|
14
|
+
- **Strategy Selection**: When to use `simple`, `parallel`, `sequential`, `doublePass`, etc.
|
|
15
|
+
- **Schema Definition**: JSON Schema patterns and shorthand field syntax
|
|
16
|
+
- **CLI Commands**: All struktur CLI commands and options
|
|
17
|
+
- **Best Practices**: Token budgets, validation retries, merge rules
|
|
18
|
+
|
|
19
|
+
## Installation
|
|
20
|
+
|
|
21
|
+
### For Claude Code
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
# Install globally
|
|
25
|
+
mkdir -p ~/.config/claude/skills
|
|
26
|
+
cp -r . ~/.config/claude/skills/struktur
|
|
27
|
+
|
|
28
|
+
# Or install in a project
|
|
29
|
+
mkdir -p .agents/skills
|
|
30
|
+
cp -r . .agents/skills/struktur
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
### For OpenCode
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
# Install globally
|
|
37
|
+
mkdir -p ~/.config/opencode/skills
|
|
38
|
+
cp -r . ~/.config/opencode/skills/struktur
|
|
39
|
+
|
|
40
|
+
# Or install in a project
|
|
41
|
+
mkdir -p .agents/skills
|
|
42
|
+
cp -r . .agents/skills/struktur
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
### For OpenAI Codex
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
# Install globally
|
|
49
|
+
mkdir -p ~/.codex/skills
|
|
50
|
+
cp -r . ~/.codex/skills/struktur
|
|
51
|
+
|
|
52
|
+
# Or install in a project
|
|
53
|
+
mkdir -p .codex/skills
|
|
54
|
+
cp -r . .codex/skills/struktur
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### For Amp
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
# Install globally
|
|
61
|
+
mkdir -p ~/.config/amp/skills
|
|
62
|
+
cp -r . ~/.config/amp/skills/struktur
|
|
63
|
+
|
|
64
|
+
# Or install in a project
|
|
65
|
+
mkdir -p .agents/skills
|
|
66
|
+
cp -r . .agents/skills/struktur
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
### For VS Code (GitHub Copilot)
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
# Install in workspace
|
|
73
|
+
mkdir -p .vscode/skills
|
|
74
|
+
cp -r . .vscode/skills/struktur
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
### For Cursor
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
# Install in project
|
|
81
|
+
mkdir -p .cursor/skills
|
|
82
|
+
cp -r . .cursor/skills/struktur
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
### Using the Skills CLI (Universal)
|
|
86
|
+
|
|
87
|
+
If you have the [skills CLI](https://github.com/vercel/skills) installed:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
# Install from npm
|
|
91
|
+
npx skills add @struktur/skill
|
|
92
|
+
|
|
93
|
+
# Or install from GitHub
|
|
94
|
+
npx skills add https://github.com/mateffy/struktur/tree/main/packages/skill/skills/struktur
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Usage
|
|
98
|
+
|
|
99
|
+
Once installed, just ask your AI agent:
|
|
100
|
+
|
|
101
|
+
```
|
|
102
|
+
"Use struktur to extract product data from this PDF"
|
|
103
|
+
"Set up struktur to parse invoices with parallel strategy"
|
|
104
|
+
"Help me configure the struktur CLI for my project"
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
The agent will automatically load this skill and follow Struktur's best practices.
|
|
108
|
+
|
|
109
|
+
## Supported Tools
|
|
110
|
+
|
|
111
|
+
This skill works with any tool that supports the [Agent Skills open standard](https://github.com/anthropics/skills):
|
|
112
|
+
|
|
113
|
+
- ✅ Claude Code
|
|
114
|
+
- ✅ OpenCode
|
|
115
|
+
- ✅ OpenAI Codex
|
|
116
|
+
- ✅ Amp
|
|
117
|
+
- ✅ VS Code (GitHub Copilot)
|
|
118
|
+
- ✅ Cursor
|
|
119
|
+
- ✅ Gemini CLI
|
|
120
|
+
- ✅ JetBrains (Junie)
|
|
121
|
+
- ✅ And 10+ more tools
|
|
122
|
+
|
|
123
|
+
## Skill Structure
|
|
124
|
+
|
|
125
|
+
```
|
|
126
|
+
struktur/
|
|
127
|
+
├── SKILL.md # Main skill file (this directory)
|
|
128
|
+
├── README.md # This file
|
|
129
|
+
├── scripts/ # (optional) executable helpers
|
|
130
|
+
├── references/ # (optional) additional docs
|
|
131
|
+
└── assets/ # (optional) templates and examples
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
## Progressive Disclosure
|
|
135
|
+
|
|
136
|
+
The skill uses progressive disclosure to minimize token usage:
|
|
137
|
+
|
|
138
|
+
1. **Metadata Phase** (~100 tokens): Agent loads only name + description
|
|
139
|
+
2. **Activation Phase**: When task matches description, agent asks permission to load
|
|
140
|
+
3. **Full Load** (up to 5,000 tokens): Complete SKILL.md loads into context
|
|
141
|
+
4. **On-Demand**: Supporting files (scripts, references) load as needed
|
|
142
|
+
|
|
143
|
+
## Resources
|
|
144
|
+
|
|
145
|
+
- [Struktur Repository](https://github.com/mateffy/struktur)
|
|
146
|
+
- [Struktur Documentation](https://github.com/mateffy/struktur#readme)
|
|
147
|
+
- [Agent Skills Specification](https://github.com/anthropics/skills)
|
|
148
|
+
- [Vercel AI SDK](https://sdk.vercel.ai)
|
|
149
|
+
|
|
150
|
+
## License
|
|
151
|
+
|
|
152
|
+
MIT
|
|
@@ -0,0 +1,404 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: struktur
|
|
3
|
+
description: Extracts structured JSON from documents using an autonomous agent with the Vercel AI SDK. The agent explores documents using a virtual filesystem (read, grep, find, bash tools) and extracts data incrementally. Also supports traditional strategies (simple/parallel/sequential/doublePass) for specific use cases. Use when working with @struktur/sdk — importing extract(), choosing extraction strategies, building Artifacts, defining schemas, or using the struktur CLI.
|
|
4
|
+
metadata:
|
|
5
|
+
author: mateffy
|
|
6
|
+
version: "2.0"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Struktur
|
|
10
|
+
|
|
11
|
+
Struktur extracts structured JSON from documents using an **autonomous agent**. The agent explores documents via a virtual filesystem, deciding when to read files, search for patterns, and build output incrementally.
|
|
12
|
+
|
|
13
|
+
**Key concept**: The `agent` strategy is the default. It works best for most documents. Traditional strategies (`simple`, `parallel`, `sequential`, `doublePass`) are available for specific cases where you need more control.
|
|
14
|
+
|
|
15
|
+
**Package**: `@struktur/sdk`
|
|
16
|
+
**Runtime**: Bun (use `bun install`, `bun test`, `bun run`)
|
|
17
|
+
|
|
18
|
+
## Core API
|
|
19
|
+
|
|
20
|
+
```typescript
|
|
21
|
+
import { extract, agent } from "@struktur/sdk";
|
|
22
|
+
import type { JSONSchemaType } from "ajv";
|
|
23
|
+
|
|
24
|
+
const result = await extract({
|
|
25
|
+
artifacts, // Artifact[] — required
|
|
26
|
+
schema, // JSONSchemaType<T> | AnyJSONSchema — mutually exclusive with fields
|
|
27
|
+
fields, // string — shorthand, e.g. "title, price:number" — mutually exclusive with schema
|
|
28
|
+
strategy, // ExtractionStrategy<T> — required (use agent() for default)
|
|
29
|
+
events?, // ExtractionEvents — optional progress hooks
|
|
30
|
+
debug?, // DebugLogger — optional JSON logging
|
|
31
|
+
strict?, // boolean — strict schema validation
|
|
32
|
+
});
|
|
33
|
+
|
|
34
|
+
result.data // T — extracted, validated output
|
|
35
|
+
result.usage // { inputTokens, outputTokens, totalTokens }
|
|
36
|
+
result.error // Error | undefined
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## Agent Strategy (Default)
|
|
40
|
+
|
|
41
|
+
The agent strategy is the **default and recommended** way to use Struktur. It gives the LLM a virtual filesystem and lets it autonomously extract your data.
|
|
42
|
+
|
|
43
|
+
### How it works
|
|
44
|
+
1. Document loaded into virtual filesystem (`/artifacts/artifact.json`, `/artifacts/images/`)
|
|
45
|
+
2. Agent explores using tools: **read**, **grep**, **find**, **ls**, **bash**
|
|
46
|
+
3. Builds output incrementally via **set_output_data** and **update_output_data**
|
|
47
|
+
4. Calls **finish** when complete, or **fail** if impossible
|
|
48
|
+
5. Schema validation on every update with automatic retry
|
|
49
|
+
|
|
50
|
+
### Configuration
|
|
51
|
+
|
|
52
|
+
```typescript
|
|
53
|
+
import { extract, agent } from "@struktur/sdk";
|
|
54
|
+
|
|
55
|
+
strategy: agent({
|
|
56
|
+
provider: "anthropic", // Provider name (anthropic, openai, google, opencode)
|
|
57
|
+
modelId: "claude-sonnet-4", // Model identifier
|
|
58
|
+
maxSteps?: 50, // Max agent steps (default: 50)
|
|
59
|
+
apiKey?: "sk-...", // API key (or use env vars)
|
|
60
|
+
outputInstructions?: "string", // Extra extraction guidance
|
|
61
|
+
})
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
### Model compatibility
|
|
65
|
+
The agent requires models with tool/function calling support:
|
|
66
|
+
- **Anthropic**: Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Haiku
|
|
67
|
+
- **OpenAI**: GPT-4o, GPT-4 Turbo, GPT-4, GPT-3.5 Turbo
|
|
68
|
+
- **Google**: Gemini 1.5 Pro, Gemini 1.5 Flash
|
|
69
|
+
|
|
70
|
+
### Example
|
|
71
|
+
|
|
72
|
+
```typescript
|
|
73
|
+
import { extract, agent } from "@struktur/sdk";
|
|
74
|
+
|
|
75
|
+
const result = await extract({
|
|
76
|
+
artifacts,
|
|
77
|
+
schema,
|
|
78
|
+
strategy: agent({
|
|
79
|
+
provider: "anthropic",
|
|
80
|
+
modelId: "claude-sonnet-4",
|
|
81
|
+
maxSteps: 50,
|
|
82
|
+
}),
|
|
83
|
+
});
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
### CLI usage
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
# Agent is the default — no --strategy needed
|
|
90
|
+
struktur extract --input ./document.pdf --schema ./schema.json
|
|
91
|
+
|
|
92
|
+
# With specific model and max steps
|
|
93
|
+
struktur extract --input ./document.pdf \
|
|
94
|
+
--schema ./schema.json \
|
|
95
|
+
--model anthropic/claude-sonnet-4 \
|
|
96
|
+
--max-steps 30
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
## Traditional Strategies
|
|
100
|
+
|
|
101
|
+
For specific use cases where you need more control over chunking and parallelism.
|
|
102
|
+
|
|
103
|
+
| Strategy | When to use | Key config |
|
|
104
|
+
|---|---|---|
|
|
105
|
+
| `agent` (default) | **Most documents** — autonomous exploration | `provider`, `modelId`, `maxSteps` |
|
|
106
|
+
| `simple` | Small input, fits in one context window | `model` |
|
|
107
|
+
| `parallel` | Large input, array/list output, order doesn't matter | `model`, `mergeModel`, `chunkSize`, `concurrency?` |
|
|
108
|
+
| `sequential` | Large input, context must carry over between batches | `model`, `chunkSize` |
|
|
109
|
+
| `parallelAutoMerge` | Large input, array output, need dedup | `model`, `chunkSize`, `dedupeModel?` |
|
|
110
|
+
| `sequentialAutoMerge` | Large input, sequential + dedup | `model`, `chunkSize`, `dedupeModel?` |
|
|
111
|
+
| `doublePass` | Max accuracy: parallel merge then sequential refinement | `model`, `mergeModel`, `chunkSize` |
|
|
112
|
+
| `doublePassAutoMerge` | Max accuracy + dedup | `model`, `chunkSize`, `dedupeModel?` |
|
|
113
|
+
|
|
114
|
+
```typescript
|
|
115
|
+
import { extract, simple, parallel, sequential, parallelAutoMerge, doublePass } from "@struktur/sdk";
|
|
116
|
+
import { google } from "@ai-sdk/google";
|
|
117
|
+
import { anthropic } from "@ai-sdk/anthropic";
|
|
118
|
+
import { openai } from "@ai-sdk/openai";
|
|
119
|
+
|
|
120
|
+
// Simple — small inputs
|
|
121
|
+
strategy: simple({ model: google("gemini-2.0-flash") })
|
|
122
|
+
|
|
123
|
+
// Parallel — large inputs, list output
|
|
124
|
+
strategy: parallel({
|
|
125
|
+
model: google("gemini-2.0-flash"),
|
|
126
|
+
mergeModel: google("gemini-2.0-flash"),
|
|
127
|
+
chunkSize: 10_000,
|
|
128
|
+
concurrency: 4,
|
|
129
|
+
})
|
|
130
|
+
|
|
131
|
+
// Sequential — context carryover between batches
|
|
132
|
+
strategy: sequential({ model: anthropic("claude-sonnet-4-5"), chunkSize: 8_000 })
|
|
133
|
+
|
|
134
|
+
// Double pass — highest accuracy
|
|
135
|
+
strategy: doublePass({
|
|
136
|
+
model: openai("gpt-4o"),
|
|
137
|
+
mergeModel: openai("gpt-4o"),
|
|
138
|
+
chunkSize: 12_000,
|
|
139
|
+
concurrency: 3,
|
|
140
|
+
})
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
**Common config options** (all strategies):
|
|
144
|
+
- `model` — Vercel AI SDK model instance
|
|
145
|
+
- `chunkSize` — token budget per batch (default: 10_000)
|
|
146
|
+
- `maxImages?` — max images per batch
|
|
147
|
+
- `outputInstructions?` — extra extraction guidance in system prompt
|
|
148
|
+
- `strict?` — strict schema validation
|
|
149
|
+
|
|
150
|
+
## Schema Definition
|
|
151
|
+
|
|
152
|
+
Two approaches — mutually exclusive:
|
|
153
|
+
|
|
154
|
+
### JSON Schema (typed)
|
|
155
|
+
```typescript
|
|
156
|
+
import type { JSONSchemaType } from "ajv";
|
|
157
|
+
|
|
158
|
+
type Output = {
|
|
159
|
+
title: string;
|
|
160
|
+
items: Array<{ name: string; price: number }>;
|
|
161
|
+
};
|
|
162
|
+
|
|
163
|
+
const schema: JSONSchemaType<Output> = {
|
|
164
|
+
type: "object",
|
|
165
|
+
properties: {
|
|
166
|
+
title: { type: "string" },
|
|
167
|
+
items: {
|
|
168
|
+
type: "array",
|
|
169
|
+
items: {
|
|
170
|
+
type: "object",
|
|
171
|
+
properties: {
|
|
172
|
+
name: { type: "string" },
|
|
173
|
+
price: { type: "number" },
|
|
174
|
+
},
|
|
175
|
+
required: ["name", "price"],
|
|
176
|
+
additionalProperties: false,
|
|
177
|
+
},
|
|
178
|
+
},
|
|
179
|
+
},
|
|
180
|
+
required: ["title", "items"],
|
|
181
|
+
additionalProperties: false,
|
|
182
|
+
};
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
### Shorthand fields string
|
|
186
|
+
```typescript
|
|
187
|
+
// "name" defaults to string; "name:type" for explicit types
|
|
188
|
+
const fields = "title, price:number, active:boolean, tags:array";
|
|
189
|
+
// Supported: string, number, boolean, integer, int, float, bool, enum{a|b|c}, array (defaults to array{string}), array{type}
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
## Building Artifacts
|
|
193
|
+
|
|
194
|
+
Struktur does NOT parse PDFs, HTML, or files — it expects pre-parsed Artifacts.
|
|
195
|
+
|
|
196
|
+
```typescript
|
|
197
|
+
import type { Artifact, ArtifactContent } from "@struktur/sdk";
|
|
198
|
+
|
|
199
|
+
// Manual construction
|
|
200
|
+
const artifact: Artifact = {
|
|
201
|
+
id: "doc-1",
|
|
202
|
+
type: "text", // "text" | "image" | "pdf" | "file"
|
|
203
|
+
raw: async () => Buffer.from("source content"),
|
|
204
|
+
contents: [
|
|
205
|
+
{ page: 1, text: "Extracted text content here..." },
|
|
206
|
+
{ page: 2, text: "More content...", media: [{ type: "image", url: "..." }] },
|
|
207
|
+
],
|
|
208
|
+
};
|
|
209
|
+
|
|
210
|
+
// From URL (fetches Artifact JSON)
|
|
211
|
+
import { urlToArtifact } from "@struktur/sdk";
|
|
212
|
+
const artifact = await urlToArtifact("https://example.com/artifact.json");
|
|
213
|
+
|
|
214
|
+
// From file buffer with custom provider
|
|
215
|
+
import { fileToArtifact } from "@struktur/sdk";
|
|
216
|
+
const providers = {
|
|
217
|
+
"application/pdf": async (buffer: Buffer) => ({
|
|
218
|
+
id: "pdf-1",
|
|
219
|
+
type: "pdf" as const,
|
|
220
|
+
raw: async () => buffer,
|
|
221
|
+
contents: [{ page: 1, text: "..." }],
|
|
222
|
+
}),
|
|
223
|
+
};
|
|
224
|
+
const artifact = await fileToArtifact(buffer, { mimeType: "application/pdf", providers });
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
**ArtifactImage** fields: `type`, `url?`, `base64?`, `contents?` (Buffer), `text?`, `x?`, `y?`, `width?`, `height?`
|
|
228
|
+
|
|
229
|
+
## Events
|
|
230
|
+
|
|
231
|
+
```typescript
|
|
232
|
+
const result = await extract({
|
|
233
|
+
artifacts,
|
|
234
|
+
schema,
|
|
235
|
+
strategy,
|
|
236
|
+
events: {
|
|
237
|
+
onStep: ({ step, total, label, detail }) => {
|
|
238
|
+
console.log(`[${step}/${total}] ${label} ${detail || ""}`);
|
|
239
|
+
// label examples: "extract", "Read manifest.json", "Set Output", "agent_explore"
|
|
240
|
+
},
|
|
241
|
+
onRetry: ({ attempt, maxAttempts, reason }) => {
|
|
242
|
+
console.warn(`Retry ${attempt}/${maxAttempts}: ${reason}`);
|
|
243
|
+
},
|
|
244
|
+
onTokenUsage: ({ inputTokens, outputTokens, totalTokens, model }) => {
|
|
245
|
+
console.log(`Tokens: ${totalTokens} (${model})`);
|
|
246
|
+
},
|
|
247
|
+
onMessage: ({ role, content }) => { /* full LLM messages */ },
|
|
248
|
+
onProgress: ({ current, total, percent }) => { /* batch progress */ },
|
|
249
|
+
},
|
|
250
|
+
});
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
## Custom Strategy
|
|
254
|
+
|
|
255
|
+
```typescript
|
|
256
|
+
import type { ExtractionStrategy, ExtractionOptions, ExtractionResult } from "@struktur/sdk";
|
|
257
|
+
|
|
258
|
+
const myStrategy: ExtractionStrategy<Output> = {
|
|
259
|
+
name: "my-strategy",
|
|
260
|
+
getEstimatedSteps: (artifacts) => 3,
|
|
261
|
+
async run(options: ExtractionOptions<Output>): Promise<ExtractionResult<Output>> {
|
|
262
|
+
return {
|
|
263
|
+
data: { /* ... */ },
|
|
264
|
+
usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 },
|
|
265
|
+
};
|
|
266
|
+
},
|
|
267
|
+
};
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
## CLI
|
|
271
|
+
|
|
272
|
+
```bash
|
|
273
|
+
# Extract with agent (default) — no --strategy needed
|
|
274
|
+
struktur extract --input ./document.pdf --schema ./schema.json
|
|
275
|
+
|
|
276
|
+
# Extract with specific model
|
|
277
|
+
struktur extract -i document.txt -s schema.json -m anthropic/claude-sonnet-4
|
|
278
|
+
|
|
279
|
+
# Extract with max steps limit
|
|
280
|
+
struktur extract -i docs.pdf -s schema.json -m anthropic/claude-sonnet-4 --max-steps 30
|
|
281
|
+
|
|
282
|
+
# Extract with inline text
|
|
283
|
+
struktur extract -t "Invoice #1234, total $99" -f "invoice_number, total:number"
|
|
284
|
+
|
|
285
|
+
# Use traditional strategy (for specific cases)
|
|
286
|
+
struktur extract -i docs.txt -s schema.json -S parallel --chunk-size 8000
|
|
287
|
+
|
|
288
|
+
# Pipe stdin
|
|
289
|
+
cat document.txt | struktur extract -f "title, summary" -m anthropic/claude-sonnet-4
|
|
290
|
+
|
|
291
|
+
# Debug mode (verbose JSON logs to stderr)
|
|
292
|
+
struktur extract --debug -t "text" -s schema.json
|
|
293
|
+
|
|
294
|
+
# Model management
|
|
295
|
+
struktur models list
|
|
296
|
+
struktur models use anthropic/claude-sonnet-4 # set default model
|
|
297
|
+
struktur models alias set fast anthropic/claude-haiku-4
|
|
298
|
+
struktur models alias set smart anthropic/claude-opus-4
|
|
299
|
+
struktur extract -t "text" -f "title" -m fast # use alias
|
|
300
|
+
|
|
301
|
+
# Provider setup
|
|
302
|
+
struktur providers add anthropic --token sk-ant-...
|
|
303
|
+
struktur providers add openai --token sk-...
|
|
304
|
+
struktur providers list
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
**Model format**: `provider/model-name`
|
|
308
|
+
**Providers**: `anthropic`, `openai`, `google`, `opencode`, `openrouter`
|
|
309
|
+
**OpenRouter provider routing**: `openrouter/anthropic/claude-3.5-sonnet#cerebras`
|
|
310
|
+
|
|
311
|
+
## Agent TUI
|
|
312
|
+
|
|
313
|
+
When using the agent strategy in the CLI, you'll see a live TUI showing:
|
|
314
|
+
|
|
315
|
+
```
|
|
316
|
+
◈ Read manifest.json
|
|
317
|
+
◉ Grep "property" in artifact.json
|
|
318
|
+
◐ Set Output {"real_estate_property":{"name":"Dock 100"...
|
|
319
|
+
◆ Bash: head -20 /artifacts/artifact.json
|
|
320
|
+
▸ Now let me add the building details
|
|
321
|
+
◑ Update Output {"buildings":[...
|
|
322
|
+
⠋ Agent: completing...
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
**Icons:**
|
|
326
|
+
- ▸ model thinking/exploring
|
|
327
|
+
- ◈ read file / agent lifecycle
|
|
328
|
+
- ◆ bash command
|
|
329
|
+
- ◉ grep search
|
|
330
|
+
- ◊ find files
|
|
331
|
+
- ◇ list directory
|
|
332
|
+
- ◐ set output data
|
|
333
|
+
- ◑ update output data
|
|
334
|
+
- ◒ finish extraction
|
|
335
|
+
- ◓ fail extraction
|
|
336
|
+
|
|
337
|
+
## Internal Architecture
|
|
338
|
+
|
|
339
|
+
**Agent Strategy:**
|
|
340
|
+
```
|
|
341
|
+
extract()
|
|
342
|
+
→ agent.run()
|
|
343
|
+
→ VirtualFilesystem (src/agent-strategy/ArtifactFilesystem.ts)
|
|
344
|
+
→ Tool definitions (src/agent-strategy/AgentTools.ts)
|
|
345
|
+
→ Agent session (@mariozechner/pi-coding-agent)
|
|
346
|
+
→ Tool calls (read, grep, find, ls, bash)
|
|
347
|
+
→ Output updates (set_output_data, update_output_data)
|
|
348
|
+
→ Validation (schema check + retry)
|
|
349
|
+
→ finish() (return validated JSON)
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
**Traditional Strategies:**
|
|
353
|
+
```
|
|
354
|
+
extract()
|
|
355
|
+
→ strategy.run()
|
|
356
|
+
→ batchArtifacts() / splitArtifact() (src/chunking/)
|
|
357
|
+
→ buildExtractorPrompt() (src/prompts/)
|
|
358
|
+
→ runWithRetries() (src/llm/)
|
|
359
|
+
→ generateStructured() (Vercel AI SDK)
|
|
360
|
+
→ schema validation / retry
|
|
361
|
+
→ merge / dedupe (src/merge/)
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
**Key modules**:
|
|
365
|
+
- `packages/agent-strategy/` — Agent strategy with virtual filesystem
|
|
366
|
+
- `packages/sdk/src/strategies/` — Traditional strategy implementations
|
|
367
|
+
- `packages/sdk/src/artifacts/` — `fileToArtifact`, `urlToArtifact`, `parseInputToArtifacts`
|
|
368
|
+
- `packages/sdk/src/chunking/` — `ArtifactSplitter`, `ArtifactBatcher`
|
|
369
|
+
- `packages/sdk/src/llm/` — `LLMClient`, `RetryingRunner`, `models`
|
|
370
|
+
- `packages/sdk/src/prompts/` — `ExtractorPrompt`, `SequentialExtractorPrompt`, `ParallelMergerPrompt`
|
|
371
|
+
- `packages/sdk/src/merge/` — `SmartDataMerger`, `Deduplicator`
|
|
372
|
+
- `packages/sdk/src/validation/` — schema validator, `SchemaValidationError`
|
|
373
|
+
- `packages/sdk/src/auth/` — token storage, alias/default model config
|
|
374
|
+
- `packages/sdk/src/debug/` — structured JSON logger
|
|
375
|
+
|
|
376
|
+
## Patterns
|
|
377
|
+
|
|
378
|
+
**Agent strategy best practices:**
|
|
379
|
+
- Always try agent first — it's the default for a reason
|
|
380
|
+
- Use models with strong tool-calling (Claude 3.5 Sonnet, GPT-4o)
|
|
381
|
+
- Adjust `maxSteps` if the agent needs more exploration time (default: 50)
|
|
382
|
+
- Use traditional strategies only when you need predictable costs or specific chunking
|
|
383
|
+
|
|
384
|
+
**Merge rules** (SmartDataMerger): arrays concatenate, objects shallow-merge, scalars prefer newest value.
|
|
385
|
+
|
|
386
|
+
**Validation retries**: `runWithRetries` feeds validation errors back to the LLM (max 3 attempts). `onRetry` fires with `{ attempt, maxAttempts, reason }`.
|
|
387
|
+
|
|
388
|
+
**Token budget**: `chunkSize` is in tokens. Default `10_000`. Images default to 1000 tokens each. Adjust `maxImages` to control multimodal batch costs.
|
|
389
|
+
|
|
390
|
+
**Schema strict mode**: Pass `strict: true` to use OpenAI's `strictJsonSchema` — requires `additionalProperties: false` everywhere and no `$ref`.
|
|
391
|
+
|
|
392
|
+
**Fields shorthand** builds a JSON Schema automatically. Use for quick CLIs or simple schemas; use `JSONSchemaType<T>` for full type inference.
|
|
393
|
+
|
|
394
|
+
**Debug logs**: Single-line JSON to stderr. Types include `cli_init`, `artifacts_loaded`, `batching_complete`, `llm_call_start`, `llm_call_complete`, `validation_failed`, `retry`, `merge_start`, `extraction_complete`.
|
|
395
|
+
|
|
396
|
+
## Tests
|
|
397
|
+
|
|
398
|
+
```bash
|
|
399
|
+
bun test # run all tests
|
|
400
|
+
bun test packages/agent-strategy/ # agent strategy tests
|
|
401
|
+
bun test packages/sdk/src/strategies/ # traditional strategy tests
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
Tests are colocated: `foo.ts` → `foo.test.ts`. Add or update tests whenever behavior changes.
|