@game_ryo/lsji 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +15 -7
- package/src/cli.js +395 -62
- package/src/execution/budget/circuit-breaker.js +245 -0
- package/src/execution/budget/cost-tracker.js +387 -0
- package/src/execution/budget/index.js +63 -0
- package/src/execution/budget/token-counter.js +159 -0
- package/src/execution/engine.js +428 -0
- package/src/execution/hitl/approval-gate.js +210 -0
- package/src/execution/hitl/index.js +12 -0
- package/src/execution/hitl/notifier.js +151 -0
- package/src/execution/hitl/store.js +311 -0
- package/src/execution/idempotency.js +312 -0
- package/src/execution/index.js +14 -0
- package/src/index.js +80 -4
- package/src/llm/index.js +21 -0
- package/src/llm/llm-agent.js +357 -0
- package/src/llm/memory/conversation.js +271 -0
- package/src/llm/memory/episodic.js +312 -0
- package/src/llm/memory/index.js +12 -0
- package/src/llm/memory/semantic.js +324 -0
- package/src/llm/plugins/index.js +202 -0
- package/src/llm/prompt-manager.js +332 -0
- package/src/llm/providers/anthropic.js +250 -0
- package/src/llm/providers/base.js +116 -0
- package/src/llm/providers/local.js +163 -0
- package/src/llm/providers/openai.js +212 -0
- package/src/llm/tools/registry.js +342 -0
- package/src/server/index.js +416 -0
- package/src/server/ui/index.html +16 -0
- package/src/server/ui/package.json +19 -0
- package/src/server/ui/src/main.jsx +10 -0
- package/src/server/ui/src/styles.css +260 -0
- package/src/server/ui/vite.config.js +27 -0
- package/docs/README.md +0 -43
- package/docs/blog/2019-05-28-first-blog-post.mdx +0 -12
- package/docs/blog/2019-05-29-long-blog-post.mdx +0 -44
- package/docs/blog/2021-08-01-mdx-blog-post.mdx +0 -24
- package/docs/blog/2021-08-26-welcome/docusaurus-plushie-banner.jpeg +0 -0
- package/docs/blog/2021-08-26-welcome/index.mdx +0 -29
- package/docs/blog/authors.yml +0 -25
- package/docs/blog/tags.yml +0 -19
- package/docs/docs/api/agent.md +0 -151
- package/docs/docs/api/env.md +0 -133
- package/docs/docs/api/environments.md +0 -102
- package/docs/docs/api/qlearning.md +0 -138
- package/docs/docs/api/storage.md +0 -168
- package/docs/docs/architecture.md +0 -155
- package/docs/docs/cli.md +0 -210
- package/docs/docs/contributing.md +0 -162
- package/docs/docs/core-concepts.md +0 -152
- package/docs/docs/examples/advanced-training.md +0 -244
- package/docs/docs/examples/custom-environment.md +0 -198
- package/docs/docs/examples/custom-storage.md +0 -251
- package/docs/docs/getting-started.md +0 -91
- package/docs/docusaurus.config.ts +0 -149
- package/docs/package-lock.json +0 -19522
- package/docs/package.json +0 -49
- package/docs/sidebars.ts +0 -33
- package/docs/src/components/HomepageFeatures/index.tsx +0 -71
- package/docs/src/components/HomepageFeatures/styles.module.css +0 -11
- package/docs/src/css/custom.css +0 -79
- package/docs/src/pages/index.module.css +0 -23
- package/docs/src/pages/index.tsx +0 -44
- package/docs/src/pages/markdown-page.mdx +0 -7
- package/docs/static/.nojekyll +0 -0
- package/docs/static/img/docusaurus-social-card.jpg +0 -0
- package/docs/static/img/docusaurus.png +0 -0
- package/docs/static/img/favicon.ico +0 -0
- package/docs/static/img/logo.png +0 -0
- package/docs/static/img/undraw_docusaurus_mountain.svg +0 -171
- package/docs/static/img/undraw_docusaurus_react.svg +0 -170
- package/docs/static/img/undraw_docusaurus_tree.svg +0 -40
- package/docs/tsconfig.json +0 -12
- package/legacy/worker.js +0 -166
- package/legacy/wrangler.toml +0 -11
package/docs/docs/cli.md
DELETED
|
@@ -1,210 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: CLI Reference
|
|
3
|
-
description: Command-line interface for training and playing
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# CLI Reference
|
|
7
|
-
|
|
8
|
-
LSJI includes a command-line interface for training and playing without writing code.
|
|
9
|
-
|
|
10
|
-
## Installation
|
|
11
|
-
|
|
12
|
-
```bash
|
|
13
|
-
# Global install
|
|
14
|
-
npm install -g lsji
|
|
15
|
-
|
|
16
|
-
# Or use npx
|
|
17
|
-
npx lsji --help
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
## Commands
|
|
21
|
-
|
|
22
|
-
### `lsji train`
|
|
23
|
-
|
|
24
|
-
Train the agent.
|
|
25
|
-
|
|
26
|
-
```bash
|
|
27
|
-
lsji train [options]
|
|
28
|
-
```
|
|
29
|
-
|
|
30
|
-
**Options:**
|
|
31
|
-
|
|
32
|
-
| Option | Description | Default |
|
|
33
|
-
|--------|-------------|---------|
|
|
34
|
-
| `--episodes <n>` | Number of training episodes | 200 |
|
|
35
|
-
| `--pattern <0-3>` | Training pattern | 0 |
|
|
36
|
-
| `--batch-size <n>` | Database batch size | 200 |
|
|
37
|
-
| `--opponent <type>` | Opponent strategy | random |
|
|
38
|
-
| `--storage <type>` | Storage backend | sqlite |
|
|
39
|
-
| `--db-path <path>` | Database file path | ./lsji.db |
|
|
40
|
-
| `--alpha <n>` | Learning rate | 0.1 |
|
|
41
|
-
| `--gamma <n>` | Discount factor | 0.9 |
|
|
42
|
-
| `--epsilon <n>` | Exploration rate | 0.1 |
|
|
43
|
-
| `--json` | Output as JSON | false |
|
|
44
|
-
|
|
45
|
-
**Training Patterns:**
|
|
46
|
-
- `0` — Random actions
|
|
47
|
-
- `1` — Always Rock
|
|
48
|
-
- `2` — Counter previous action
|
|
49
|
-
- `3` — Sequential (0,1,2,0,1,2...)
|
|
50
|
-
|
|
51
|
-
**Opponent Strategies:**
|
|
52
|
-
- `random` — Random actions
|
|
53
|
-
- `always_rock` — Always plays Rock
|
|
54
|
-
- `counter` — Counters agent's previous action
|
|
55
|
-
- `sequential` — Cycles through actions
|
|
56
|
-
|
|
57
|
-
**Examples:**
|
|
58
|
-
```bash
|
|
59
|
-
# Default training
|
|
60
|
-
lsji train --episodes 500
|
|
61
|
-
|
|
62
|
-
# Train against always-rock opponent
|
|
63
|
-
lsji train --episodes 100 --pattern 1 --opponent always_rock
|
|
64
|
-
|
|
65
|
-
# Train with custom hyperparameters
|
|
66
|
-
lsji train --episodes 1000 --alpha 0.05 --gamma 0.95 --epsilon 0.2
|
|
67
|
-
|
|
68
|
-
# Use memory storage (ephemeral)
|
|
69
|
-
lsji train --episodes 100 --storage memory
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
### `lsji play`
|
|
73
|
-
|
|
74
|
-
Play a single game against the agent.
|
|
75
|
-
|
|
76
|
-
```bash
|
|
77
|
-
lsji play --hand <0|1|2> [options]
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
**Options:**
|
|
81
|
-
|
|
82
|
-
| Option | Description |
|
|
83
|
-
|--------|-------------|
|
|
84
|
-
| `--hand <0\|1\|2>` | Your hand: 0=Rock, 1=Scissors, 2=Paper |
|
|
85
|
-
| `--opponent <type>` | Opponent strategy |
|
|
86
|
-
| `--storage <type>` | Storage backend |
|
|
87
|
-
| `--db-path <path>` | Database file path |
|
|
88
|
-
| `--json` | Output as JSON |
|
|
89
|
-
|
|
90
|
-
**Examples:**
|
|
91
|
-
```bash
|
|
92
|
-
lsji play --hand 0 # Play Rock
|
|
93
|
-
lsji play --hand 1 # Play Scissors
|
|
94
|
-
lsji play --hand 2 --json # Play Paper, JSON output
|
|
95
|
-
```
|
|
96
|
-
|
|
97
|
-
### `lsji status`
|
|
98
|
-
|
|
99
|
-
Show system status and statistics.
|
|
100
|
-
|
|
101
|
-
```bash
|
|
102
|
-
lsji status [options]
|
|
103
|
-
```
|
|
104
|
-
|
|
105
|
-
**Options:**
|
|
106
|
-
|
|
107
|
-
| Option | Description |
|
|
108
|
-
|--------|-------------|
|
|
109
|
-
| `--storage <type>` | Storage backend |
|
|
110
|
-
| `--db-path <path>` | Database file path |
|
|
111
|
-
| `--json` | Output as JSON |
|
|
112
|
-
|
|
113
|
-
**Example:**
|
|
114
|
-
```bash
|
|
115
|
-
lsji status --json
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
Output:
|
|
119
|
-
```json
|
|
120
|
-
{
|
|
121
|
-
"status": "running",
|
|
122
|
-
"todayTotal": 42,
|
|
123
|
-
"limit": 90000,
|
|
124
|
-
"performance": [
|
|
125
|
-
{ "mode": "train", "total": 1000, "win_rate": 65.5 },
|
|
126
|
-
{ "mode": "test", "total": 50, "win_rate": 72.0 }
|
|
127
|
-
],
|
|
128
|
-
"aiBrain": [
|
|
129
|
-
{ "state": "0", "action": 0, "q_value": 0.45 },
|
|
130
|
-
{ "state": "0", "action": 1, "q_value": 0.12 }
|
|
131
|
-
]
|
|
132
|
-
}
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
### `lsji start`
|
|
136
|
-
|
|
137
|
-
Enable training and play.
|
|
138
|
-
|
|
139
|
-
```bash
|
|
140
|
-
lsji start [options]
|
|
141
|
-
```
|
|
142
|
-
|
|
143
|
-
### `lsji stop`
|
|
144
|
-
|
|
145
|
-
Disable training and play (system paused).
|
|
146
|
-
|
|
147
|
-
```bash
|
|
148
|
-
lsji stop [options]
|
|
149
|
-
```
|
|
150
|
-
|
|
151
|
-
### `lsji help`
|
|
152
|
-
|
|
153
|
-
Show help message.
|
|
154
|
-
|
|
155
|
-
```bash
|
|
156
|
-
lsji help
|
|
157
|
-
lsji --help
|
|
158
|
-
lsji -h
|
|
159
|
-
```
|
|
160
|
-
|
|
161
|
-
## Environment Variables
|
|
162
|
-
|
|
163
|
-
| Variable | Description | Default |
|
|
164
|
-
|----------|-------------|---------|
|
|
165
|
-
| `LSJI_STORAGE` | Default storage backend | sqlite |
|
|
166
|
-
| `LSJI_DB_PATH` | Default database path | ./lsji.db |
|
|
167
|
-
|
|
168
|
-
## Examples
|
|
169
|
-
|
|
170
|
-
### Full Training Session
|
|
171
|
-
|
|
172
|
-
```bash
|
|
173
|
-
# Start fresh
|
|
174
|
-
rm -f lsji.db
|
|
175
|
-
|
|
176
|
-
# Train against random opponent
|
|
177
|
-
lsji train --episodes 500 --opponent random
|
|
178
|
-
|
|
179
|
-
# Train against counter opponent
|
|
180
|
-
lsji train --episodes 500 --opponent counter
|
|
181
|
-
|
|
182
|
-
# Check progress
|
|
183
|
-
lsji status --json
|
|
184
|
-
|
|
185
|
-
# Play a few games
|
|
186
|
-
lsji play --hand 0
|
|
187
|
-
lsji play --hand 1
|
|
188
|
-
lsji play --hand 2
|
|
189
|
-
```
|
|
190
|
-
|
|
191
|
-
### Using Memory Storage (CI/Testing)
|
|
192
|
-
|
|
193
|
-
```bash
|
|
194
|
-
lsji train --episodes 100 --storage memory
|
|
195
|
-
lsji play --hand 0 --storage memory
|
|
196
|
-
lsji status --storage memory
|
|
197
|
-
```
|
|
198
|
-
|
|
199
|
-
### Custom Hyperparameters
|
|
200
|
-
|
|
201
|
-
```bash
|
|
202
|
-
lsji train \
|
|
203
|
-
--episodes 2000 \
|
|
204
|
-
--alpha 0.05 \
|
|
205
|
-
--gamma 0.95 \
|
|
206
|
-
--epsilon 0.2 \
|
|
207
|
-
--opponent random \
|
|
208
|
-
--storage sqlite \
|
|
209
|
-
--db-path ./custom.db
|
|
210
|
-
```
|
|
@@ -1,162 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: Contributing
|
|
3
|
-
description: How to contribute to LSJI
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Contributing
|
|
7
|
-
|
|
8
|
-
Thank you for your interest in contributing to LSJI!
|
|
9
|
-
|
|
10
|
-
## Development Setup
|
|
11
|
-
|
|
12
|
-
```bash
|
|
13
|
-
# Clone the repository
|
|
14
|
-
git clone https://github.com/ryotagtagtag-wq/LSJI.git
|
|
15
|
-
cd LSJI
|
|
16
|
-
|
|
17
|
-
# Install dependencies
|
|
18
|
-
npm install
|
|
19
|
-
|
|
20
|
-
# Run tests
|
|
21
|
-
npm test
|
|
22
|
-
|
|
23
|
-
# Build documentation
|
|
24
|
-
cd docs && npm run build
|
|
25
|
-
```
|
|
26
|
-
|
|
27
|
-
## Project Structure
|
|
28
|
-
|
|
29
|
-
```
|
|
30
|
-
LSJI/
|
|
31
|
-
├── src/ # Core library
|
|
32
|
-
│ ├── core/ # QLearning, Agent, Env
|
|
33
|
-
│ ├── storage/ # Storage backends
|
|
34
|
-
│ ├── envs/ # Built-in environments
|
|
35
|
-
│ ├── cli.ts # CLI
|
|
36
|
-
│ └── index.ts # Public exports
|
|
37
|
-
├── test/ # Vitest tests
|
|
38
|
-
├── docs/ # Docusaurus documentation
|
|
39
|
-
└── bin/ # CLI entry point
|
|
40
|
-
```
|
|
41
|
-
|
|
42
|
-
## Making Changes
|
|
43
|
-
|
|
44
|
-
### 1. Create a Branch
|
|
45
|
-
|
|
46
|
-
```bash
|
|
47
|
-
git checkout -b feature/my-feature
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
### 2. Make Changes
|
|
51
|
-
|
|
52
|
-
Follow the existing code style:
|
|
53
|
-
- TypeScript with JSDoc comments
|
|
54
|
-
- ESM imports/exports
|
|
55
|
-
- No external dependencies in core
|
|
56
|
-
|
|
57
|
-
### 3. Run Tests
|
|
58
|
-
|
|
59
|
-
```bash
|
|
60
|
-
npm test
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
### 4. Update Documentation
|
|
64
|
-
|
|
65
|
-
If you add new features, update relevant docs in `docs/docs/`.
|
|
66
|
-
|
|
67
|
-
### 5. Commit
|
|
68
|
-
|
|
69
|
-
```bash
|
|
70
|
-
git add .
|
|
71
|
-
git commit -m "feat: add my feature"
|
|
72
|
-
```
|
|
73
|
-
|
|
74
|
-
**Commit Message Format:**
|
|
75
|
-
- `feat:` — New feature
|
|
76
|
-
- `fix:` — Bug fix
|
|
77
|
-
- `docs:` — Documentation
|
|
78
|
-
- `refactor:` — Code refactoring
|
|
79
|
-
- `test:` — Tests
|
|
80
|
-
- `chore:` — Maintenance
|
|
81
|
-
|
|
82
|
-
### 6. Push and Create PR
|
|
83
|
-
|
|
84
|
-
```bash
|
|
85
|
-
git push origin feature/my-feature
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
## Adding a New Environment
|
|
89
|
-
|
|
90
|
-
1. Create `src/envs/my-env.ts` extending `Env`
|
|
91
|
-
2. Implement all abstract methods
|
|
92
|
-
3. Export from `src/index.ts`
|
|
93
|
-
4. Add documentation in `docs/docs/api/environments.md`
|
|
94
|
-
5. Add example in `docs/docs/examples/`
|
|
95
|
-
|
|
96
|
-
## Adding a New Storage Backend
|
|
97
|
-
|
|
98
|
-
1. Create `src/storage/my-backend.ts` extending `Storage`
|
|
99
|
-
2. Implement all abstract methods
|
|
100
|
-
3. Add to `createStorage` factory in `src/storage/index.ts`
|
|
101
|
-
3. Export from `src/index.ts`
|
|
102
|
-
4. Add tests in `test/storage/`
|
|
103
|
-
|
|
104
|
-
## Modifying Learning Algorithm
|
|
105
|
-
|
|
106
|
-
1. Extend `QLearning` class or create new class in `src/core/`
|
|
107
|
-
2. Maintain compatibility with `Agent` interface
|
|
108
|
-
3. Add tests for new algorithm
|
|
109
|
-
4. Document in `docs/docs/api/`
|
|
110
|
-
|
|
111
|
-
## Code Style
|
|
112
|
-
|
|
113
|
-
- **TypeScript** with strict mode
|
|
114
|
-
- **ESM** modules (`import`/`export`)
|
|
115
|
-
- **JSDoc** for all public APIs
|
|
116
|
-
- **No `any`** unless absolutely necessary
|
|
117
|
-
- **Async/await** for async operations
|
|
118
|
-
|
|
119
|
-
## Testing Guidelines
|
|
120
|
-
|
|
121
|
-
- Use `MemoryStorage` for unit tests
|
|
122
|
-
- Test both success and error cases
|
|
123
|
-
- Test edge cases (empty Q-table, terminal states)
|
|
124
|
-
- Keep tests fast and isolated
|
|
125
|
-
|
|
126
|
-
```typescript
|
|
127
|
-
// Example test structure
|
|
128
|
-
import { describe, it, expect, beforeEach } from 'vitest';
|
|
129
|
-
import { MyFeature } from '../src/core/my-feature';
|
|
130
|
-
import { MemoryStorage } from '../src/storage/memory';
|
|
131
|
-
|
|
132
|
-
describe('MyFeature', () => {
|
|
133
|
-
let storage;
|
|
134
|
-
let feature;
|
|
135
|
-
|
|
136
|
-
beforeEach(async () => {
|
|
137
|
-
storage = new MemoryStorage();
|
|
138
|
-
await storage.initialize();
|
|
139
|
-
feature = new MyFeature({ storage });
|
|
140
|
-
});
|
|
141
|
-
|
|
142
|
-
it('should do something', async () => {
|
|
143
|
-
const result = await feature.doSomething();
|
|
144
|
-
expect(result).toBe(expected);
|
|
145
|
-
});
|
|
146
|
-
});
|
|
147
|
-
```
|
|
148
|
-
|
|
149
|
-
## Documentation
|
|
150
|
-
|
|
151
|
-
- Update relevant `.md` files in `docs/docs/`
|
|
152
|
-
- Add JSDoc comments for new public APIs
|
|
153
|
-
- Include code examples
|
|
154
|
-
|
|
155
|
-
## License
|
|
156
|
-
|
|
157
|
-
By contributing, you agree that your contributions will be licensed under the Apache 2.0 License.
|
|
158
|
-
|
|
159
|
-
## Questions?
|
|
160
|
-
|
|
161
|
-
- Open a [GitHub Issue](https://github.com/ryotagtagtag-wq/LSJI/issues)
|
|
162
|
-
- Start a [Discussion](https://github.com/ryotagtagtag-wq/LSJI/discussions)
|
|
@@ -1,152 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: Core Concepts
|
|
3
|
-
description: Understand the core architecture of LSJI
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Core Concepts
|
|
7
|
-
|
|
8
|
-
LSJI is built around four core abstractions that work together to create a flexible reinforcement learning framework.
|
|
9
|
-
|
|
10
|
-
## Architecture Overview
|
|
11
|
-
|
|
12
|
-
```
|
|
13
|
-
┌─────────────────────────────────────────────────────────────┐
|
|
14
|
-
│ Agent │
|
|
15
|
-
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
|
|
16
|
-
│ │ QLearning │ │ Storage │ │ Env │ │
|
|
17
|
-
│ │ (Engine) │◄─┤ (Backend) │ │ (Environment) │ │
|
|
18
|
-
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
|
|
19
|
-
└─────────────────────────────────────────────────────────────┘
|
|
20
|
-
```
|
|
21
|
-
|
|
22
|
-
### 1. Environment (`Env`)
|
|
23
|
-
|
|
24
|
-
The `Env` interface defines the problem domain. Any RL environment must implement:
|
|
25
|
-
|
|
26
|
-
```typescript
|
|
27
|
-
abstract class Env {
|
|
28
|
-
getState(): string; // Current state representation
|
|
29
|
-
step(action: number): Promise<StepResult>; // Execute action
|
|
30
|
-
actionSize(): number; // Number of possible actions
|
|
31
|
-
reset(): Promise<string>; // Reset to initial state
|
|
32
|
-
}
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
**StepResult** contains:
|
|
36
|
-
- `state` — New state after action
|
|
37
|
-
- `reward` — Reward received (-1, 0, 1)
|
|
38
|
-
- `done` — Whether episode ended
|
|
39
|
-
- `info` — Additional diagnostic info
|
|
40
|
-
|
|
41
|
-
### 2. Q-Learning Engine (`QLearning`)
|
|
42
|
-
|
|
43
|
-
Tabular Q-Learning with Temporal Difference (TD) updates:
|
|
44
|
-
|
|
45
|
-
```typescript
|
|
46
|
-
class QLearning {
|
|
47
|
-
constructor({ alpha, gamma, epsilon, storage });
|
|
48
|
-
|
|
49
|
-
// Epsilon-greedy action selection
|
|
50
|
-
async act(state: string, actionSize: number): Promise<number>;
|
|
51
|
-
|
|
52
|
-
// Full TD update: Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') - Q(s,a)]
|
|
53
|
-
async learn(state, action, reward, nextState, nextActionSize);
|
|
54
|
-
|
|
55
|
-
// Simplified update (terminal states): Q(s,a) ← Q(s,a) + α[r - Q(s,a)]
|
|
56
|
-
async learnSimple(state, action, reward);
|
|
57
|
-
|
|
58
|
-
// Get all Q-values for inspection
|
|
59
|
-
async getFullQTable(): Promise<QTableRecord[]>;
|
|
60
|
-
}
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
**Hyperparameters:**
|
|
64
|
-
- `alpha` (0.1) — Learning rate
|
|
65
|
-
- `gamma` (0.9) — Discount factor
|
|
66
|
-
- `epsilon` (0.1) — Exploration rate
|
|
67
|
-
|
|
68
|
-
### 3. Storage Backend (`Storage`)
|
|
69
|
-
|
|
70
|
-
Pluggable persistence layer with three implementations:
|
|
71
|
-
|
|
72
|
-
| Backend | Package | Use Case |
|
|
73
|
-
|---------|---------|----------|
|
|
74
|
-
| `SqliteStorage` | `node:sqlite` (built-in) | **Recommended** — Zero dependencies |
|
|
75
|
-
| `BetterSqliteStorage` | `better-sqlite3` | High-performance synchronous access |
|
|
76
|
-
| `MemoryStorage` | Built-in | Testing, CI, ephemeral workloads |
|
|
77
|
-
|
|
78
|
-
All implement the same interface:
|
|
79
|
-
```typescript
|
|
80
|
-
interface Storage {
|
|
81
|
-
initialize(): Promise<void>;
|
|
82
|
-
close(): Promise<void>;
|
|
83
|
-
getSetting(key): Promise<Setting>;
|
|
84
|
-
setSetting(key, value): Promise<void>;
|
|
85
|
-
getQTable(): Promise<QTableRecord[]>;
|
|
86
|
-
updateQ(state, action, qValue): Promise<void>;
|
|
87
|
-
addBattle(record): Promise<void>;
|
|
88
|
-
getTodayBattleCount(): Promise<number>;
|
|
89
|
-
getPerformanceStats(): Promise<PerformanceStat[]>;
|
|
90
|
-
}
|
|
91
|
-
```
|
|
92
|
-
|
|
93
|
-
### 4. Agent (`Agent`)
|
|
94
|
-
|
|
95
|
-
High-level orchestration combining all components:
|
|
96
|
-
|
|
97
|
-
```typescript
|
|
98
|
-
class Agent {
|
|
99
|
-
constructor({ qlearning, storage, env });
|
|
100
|
-
|
|
101
|
-
async train({ episodes, actionSelector, batchSize });
|
|
102
|
-
async play(options?): Promise<PlayResult>;
|
|
103
|
-
async status(): Promise<StatusInfo>;
|
|
104
|
-
async start(): Promise<{status, message}>;
|
|
105
|
-
async stop(): Promise<{status, message}>;
|
|
106
|
-
setEnvironment(env): void;
|
|
107
|
-
}
|
|
108
|
-
```
|
|
109
|
-
|
|
110
|
-
## Data Flow
|
|
111
|
-
|
|
112
|
-
### Training Loop
|
|
113
|
-
```
|
|
114
|
-
for each episode:
|
|
115
|
-
1. Get current state from Env
|
|
116
|
-
2. Select action via QLearning.act() (ε-greedy)
|
|
117
|
-
3. Execute action in Env → StepResult
|
|
118
|
-
4. Update Q-table via QLearning.learnSimple()
|
|
119
|
-
5. Persist battle record to Storage
|
|
120
|
-
6. Batch DB writes for performance
|
|
121
|
-
```
|
|
122
|
-
|
|
123
|
-
### Play Loop
|
|
124
|
-
```
|
|
125
|
-
1. Get current state from Env
|
|
126
|
-
2. Select best action via QLearning.act() (ε=0 for exploitation)
|
|
127
|
-
3. Execute action in Env
|
|
128
|
-
4. Update Q-table with result
|
|
129
|
-
5. Record battle to Storage
|
|
130
|
-
6. Return result
|
|
131
|
-
```
|
|
132
|
-
|
|
133
|
-
## Reward System (RPS Example)
|
|
134
|
-
|
|
135
|
-
| Outcome | Judge Formula | Reward |
|
|
136
|
-
|---------|---------------|--------|
|
|
137
|
-
| Win | (ai - user + 3) % 3 = 2 | +1 |
|
|
138
|
-
| Lose | (ai - user + 3) % 3 = 1 | -1 |
|
|
139
|
-
| Draw | (ai - user + 3) % 3 = 0 | 0 |
|
|
140
|
-
|
|
141
|
-
## Training Patterns
|
|
142
|
-
|
|
143
|
-
Built-in patterns for the RPS environment:
|
|
144
|
-
|
|
145
|
-
| Pattern | ID | Description |
|
|
146
|
-
|---------|-----|-------------|
|
|
147
|
-
| Random | 0 | Uniform random actions |
|
|
148
|
-
| Always Rock | 1 | Always play action 0 |
|
|
149
|
-
| Counter | 2 | Play counter to previous action |
|
|
150
|
-
| Sequential | 3 | Cycle through 0,1,2,0,1,2... |
|
|
151
|
-
|
|
152
|
-
Custom patterns can be implemented via `actionSelector` function.
|