@aeye/ginny 0.3.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md ADDED
@@ -0,0 +1,330 @@
1
+ # ginny
2
+
3
+ > A CLI that turns natural-language requests into executable
4
+ > [gin](../gin) programs. Types, functions, and vars accumulate across
5
+ > sessions as JSON on disk — the LLM builds a living catalog of
6
+ > reusable code that grows with your project.
7
+
8
+ ```bash
9
+ npm install -g @aeye/ginny
10
+ cd my-project
11
+ ginny # opens an interactive REPL
12
+ ginny "add 2 and 3" # one-shot
13
+ ```
14
+
15
+ ## What ginny does
16
+
17
+ You describe what you want. A **programmer** sub-agent writes a gin
18
+ program to do it, tests it against sample inputs, and returns the
19
+ result. If it needs types, reusable functions, or named vars, it asks
20
+ specialist sub-agents that search the local catalog and create new
21
+ entries when nothing matches.
22
+
23
+ Everything is typed end-to-end. Every write/test/finish cycle happens
24
+ inside gin's type system — the agent can't produce invalid expressions,
25
+ and the structured output you get back carries full type information.
26
+
27
+ For complex requests (multiple types and functions, ambiguous scope,
28
+ "build me a small system…") the programmer pauses to ask clarifying
29
+ questions, writes a plan listing the types/fns/vars it intends to
30
+ create, and waits for your approval before any code is written. Simple
31
+ requests like "add 2 and 3" skip the dance.
32
+
33
+ ## First run
34
+
35
+ ```bash
36
+ $ cd my-new-project
37
+ $ ginny
38
+
39
+ Created /path/to/my-new-project/config.json
40
+ Added config.json + ginny.log to .gitignore
41
+
42
+ Populate the file before re-running:
43
+ At least one AI provider:
44
+ - OPENAI_API_KEY (openai)
45
+ - OPENROUTER_API_KEY (openrouter)
46
+ - AWS Bedrock — any valid AWS credential source works (env vars,
47
+ `aws sso login`, IAM role, ~/.aws/credentials, etc.). Ginny
48
+ probes the credential chain at startup; AWS_REGION optional.
49
+ TAVILY_API_KEY — optional, enables web_search tool
50
+ GIN_PROVIDER — optional, preferred provider (openai | openrouter | aws)
51
+ GIN_MODEL — optional, specific model id
52
+ GIN_SEARCH_THRESHOLD — optional, corpus size below which search returns all (default 20)
53
+ GIN_TOOL_ITERATIONS — optional, max tool-call iterations per prompt run (default 100)
54
+
55
+ Environment variables still win over config.json values.
56
+ ```
57
+
58
+ Edit `config.json`, set at least one provider key, and re-run:
59
+
60
+ ```json
61
+ {
62
+ "OPENAI_API_KEY": "sk-...",
63
+ "GIN_PROVIDER": "openai",
64
+ "GIN_MODEL": "gpt-4o-mini",
65
+ "TAVILY_API_KEY": "tvly-..."
66
+ }
67
+ ```
68
+
69
+ ## Architecture
70
+
71
+ ginny is a small council of sub-agents, each specialized:
72
+
73
+ ```
74
+ ┌─────────────┐
75
+ user request ──▶ │ programmer │
76
+ └──────┬──────┘
77
+ ┌────────────────┬──────┴──────┬────────────────┐
78
+ ▼ ▼ ▼ ▼
79
+ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
80
+ │ architect │ │ designer │ │ dba │ │ researcher │
81
+ │ (types) │ │ (fns) │ │ (vars) │ │ (web search │
82
+ │ │ │ │ │ │ │ + pages) │
83
+ └─────────────┘ └──────┬───────┘ └──────────────┘ └──────────────┘
84
+ │
85
+ ▼ (recursive spin-up)
86
+ programmer
87
+ ```
88
+
89
+ - **programmer** — writes a gin `ExprDef`, calls `test()` against it,
90
+ and `finish()` when a test passes. Has the build tools
91
+ (`write` / `test` / `finish`), the find-or-create tools for pulling
92
+ in catalog items, an `edit_type` tool for backwards-compatible type
93
+ edits, and a `research` tool for factual lookups.
94
+ - **architect** — searches `./types/*.json` by keyword (top-N when the
95
+ catalog grows past the threshold, or all entries below); returns
96
+ existing types or designs new ones.
97
+ - **designer** — same pattern over `./fns/*.json`. Has both
98
+ `create_new_fn` (new function from scratch — recursively spawns a
99
+ programmer to author the body) and `edit_fn` (backwards-compatible
100
+ signature change + fresh body). The compat checker accepts widening
101
+ edits and rejects breaking ones.
102
+ - **dba** — same pattern over `./vars/*.json` (typed named values the
103
+ user or agent can read/write).
104
+ - **researcher** — wraps `web_search` + `web_get_page`; answers a
105
+ natural-language question iteratively and returns
106
+ `{ answer, sources }`.
107
+
108
+ ## Persistence
109
+
110
+ Every catalog entry is one JSON file per name. The filename IS the
111
+ identity. The three directories are relative to your current working
112
+ directory:
113
+
114
+ ```
115
+ ./types/Task.json # the Task type
116
+ ./fns/factorial.json # the factorial function
117
+ ./vars/apiBaseUrl.json # a persistent var (type + value + docs)
118
+ ```
119
+
120
+ You can hand-edit any of these between sessions. The next run picks up
121
+ your changes. Drop a new file into any of the three directories by
122
+ hand and ginny discovers it on the next search.
123
+
124
+ ### Example: `./vars/apiBaseUrl.json`
125
+
126
+ A var is a `{type, value, docs}` triple — the simplest on-disk shape:
127
+
128
+ ```json
129
+ {
130
+ "type": { "name": "text", "options": { "pattern": "^https?://" } },
131
+ "value": "https://api.example.com",
132
+ "docs": "production API root"
133
+ }
134
+ ```
135
+
136
+ Loaded at use time, `vars.apiBaseUrl` shows up in scope as a typed
137
+ `text` value that any program can read.
138
+
139
+ ### Types and functions
140
+
141
+ `./types/<Name>.json` is a `TypeDef` — gin's serialized type
142
+ descriptor. `./fns/<name>.json` is a `function`-typed `TypeDef` whose
143
+ body lives at `call.get` (gin's native callable shape — see
144
+ [gin/src/path.ts](../gin/src/path.ts) for how the path walker
145
+ dispatches). The top-level `docs` field is the function's description.
146
+
147
+ See the [gin README](../gin#core-concepts) for what TypeDef and ExprDef
148
+ look like.
149
+
150
+ ## Built-in globals
151
+
152
+ Programs always have access to:
153
+
154
+ - **`fns.fetch<R: any>({ url, method?, headers?, body?, output?: typ<R> }): R`**
155
+ HTTP fetch. When `output` is a gin Type, the response body is parsed
156
+ through it — type-safe HTTP in one call.
157
+
158
+ - **`fns.llm<R: text | obj>({ prompt, tools?, output?: typ<R> }): R`**
159
+ LLM invocation. Pass a gin Type as `output` to get structured, typed
160
+ output. The `<R: text | obj>` constraint says R must be either a
161
+ text-shaped reply or an obj-shaped structured output — chosen at the
162
+ call site.
163
+
164
+ - **`fns.log({ message: any }): void`**
165
+ Print a runtime message to the user (stderr). Use for progress
166
+ narration, intermediate values, or debug breadcrumbs. Distinct from
167
+ the program's return value.
168
+
169
+ - **`fns.ask<R: any>({ title: text, details: text, output?: typ<R> }): optional<R>`**
170
+ Pause the program and prompt the user. With `output` set the
171
+ consumer walks the user through any complex shape (obj fields, list
172
+ items, choices, optionals) — every (sub)type's `docs` field becomes
173
+ the user-facing label. Returns `null` (`optional<R>`) if the user
174
+ cancels, so the program must handle that branch explicitly.
175
+
176
+ - **`vars.<name>`** — any var you've created or imported.
177
+
178
+ ## The write / test / finish loop
179
+
180
+ ```
181
+ > compute the factorial of 6
182
+
183
+ • (programmer calls find_or_create_functions "factorial function")
184
+ • (designer spins up a fresh programmer → writes the recursive gin program)
185
+ • (programmer calls write(program))
186
+ • (programmer calls test() → SUCCESS: 720)
187
+ • (programmer calls finish())
188
+
189
+ 720
190
+ ```
191
+
192
+ `test()` runs the draft program against sample args. The programmer
193
+ can set `expectError: true` to verify a program raises — useful for
194
+ "divide 1 by 0 and tell me what happens".
195
+
196
+ `finish()` accepts an optional `saveAs: '<camelCaseName>'` to persist
197
+ the program as a reusable function — every saved fn becomes a
198
+ callable global, so subsequent runs can invoke it directly.
199
+
200
+ ## Editing existing types and functions
201
+
202
+ Two tools cover backwards-compatible edits:
203
+
204
+ - **`edit_type({ name, def })`** (programmer) — replace a saved type's
205
+ definition. Allowed: add OPTIONAL fields, widen existing field
206
+ types, loosen constraints. Rejected: remove fields, add required
207
+ fields, narrow field types, change the type class.
208
+ - **`edit_fn({ name, args, returns, body })`** (designer) — change a
209
+ saved function's signature and body. Args may add optional params
210
+ or widen existing param types; returns may NARROW. The body is
211
+ rewritten from scratch by an inner programmer.
212
+
213
+ Both tools enforce backwards-compat at parse time and reject breaking
214
+ changes with a structured error. If a change is genuinely incompatible
215
+ the right move is usually to create a new type / fn under a different
216
+ name so existing callers keep working.
217
+
218
+ ## Configuration
219
+
220
+ Config values can come from `config.json` in the current working
221
+ directory, or from environment variables (env wins on conflict):
222
+
223
+ | Key | Purpose |
224
+ |---|---|
225
+ | `OPENAI_API_KEY` | enables OpenAI provider |
226
+ | `OPENROUTER_API_KEY` | enables OpenRouter provider |
227
+ | `AWS_REGION` | region for AWS Bedrock (default `us-east-1`) |
228
+ | `TAVILY_API_KEY` | enables the `web_search` tool |
229
+ | `GIN_PROVIDER` | preferred provider (openai \| openrouter \| aws) |
230
+ | `GIN_MODEL` | pin a specific model id (fallback for any sub-agent without an override) |
231
+ | `GIN_PROGRAMMER_MODEL` | model id for the programmer sub-agent |
232
+ | `GIN_DESIGNER_MODEL` | model id for the designer (fns) sub-agent |
233
+ | `GIN_ARCHITECT_MODEL` | model id for the architect (types) sub-agent |
234
+ | `GIN_DBA_MODEL` | model id for the dba (vars) sub-agent |
235
+ | `GIN_RESEARCHER_MODEL` | model id for the researcher sub-agent |
236
+ | `GIN_LLM_MODEL` | model id for the in-program `fns.llm` calls |
237
+ | `GIN_SEARCH_THRESHOLD` | corpus size below which catalog search returns all entries (default 20) |
238
+ | `GIN_TOOL_ITERATIONS` | max tool-call iterations per prompt run (default 100) |
239
+
240
+ ### AWS Bedrock
241
+
242
+ AWS isn't behind a single env var. Ginny probes the AWS SDK's standard
243
+ credential chain at startup — if it can call `ListFoundationModels`,
244
+ Bedrock is added as a provider. That means **any** of these work
245
+ without extra config:
246
+
247
+ - `aws sso login` (SSO profile active in the current shell)
248
+ - `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` env vars
249
+ - IAM role attached to the EC2/ECS/Lambda/etc. instance
250
+ - `~/.aws/credentials` with a default profile
251
+ - Container credential provider
252
+
253
+ At startup ginny prints a line like:
254
+
255
+ ```
256
+ ginny: providers enabled → openai, aws + web_search (tavily)
257
+ skipped → openrouter (OPENROUTER_API_KEY unset)
258
+ ```
259
+
260
+ At least one provider must resolve. Tavily is optional — without it
261
+ the programmer still has `web_get_page` (fetch + strip HTML).
262
+
263
+ ## Logging
264
+
265
+ Each session writes a verbose timeline to `./ginny.log` (truncated on
266
+ startup). Tool inputs and outputs, full validation problems, full
267
+ zod parse errors, and stack traces all land in the log; the terminal
268
+ view stays compact (one line per error, capped at 200–4096 chars
269
+ depending on context). When something goes sideways, `ginny.log` is
270
+ where to look.
271
+
272
+ ## Example sessions
273
+
274
+ ```
275
+ > fetch the title of example.com
276
+ → reads web_get_page, extracts <title>, returns the text.
277
+
278
+ > remember my api base url is https://api.example.com as 'apiBaseUrl'
279
+ → vars manager creates ./vars/apiBaseUrl.json
280
+
281
+ > print my api base url
282
+ → programmer reads vars.apiBaseUrl, returns the string.
283
+
284
+ > define a Task type with title, done, due
285
+ → architect creates ./types/Task.json (extending obj with props).
286
+
287
+ > create a program that counts done tasks from a list of tasks
288
+ → programmer emits a list.filter + .length program using Task.
289
+
290
+ > add an `assignee` field to Task (optional)
291
+ → programmer calls edit_type — backwards-compatible widening accepted.
292
+ ```
293
+
294
+ ## Building from source
295
+
296
+ ```bash
297
+ git clone https://github.com/ClickerMonkey/aeye.git
298
+ cd aeye
299
+ npm install
300
+ cd packages/ginny
301
+ npm run start # dev (tsx --conditions=source)
302
+ npm run build # bundled dist/index.js with shebang
303
+ ```
304
+
305
+ The production build is a single ESM file with a Node shebang — the
306
+ global install (`npm i -g @aeye/ginny`) links `ginny` straight to it.
307
+
308
+ ## How it relates to gin
309
+
310
+ ginny is an application built on top of [`@aeye/gin`](../gin). gin
311
+ provides:
312
+
313
+ - the type system (`num`, `list<V>`, `typ<T>`, user extensions, ...)
314
+ - the expression engine (`engine.run(expr)`)
315
+ - the Zod schema generation the LLM uses to author valid programs
316
+
317
+ ginny provides:
318
+
319
+ - the AI wiring (provider selection, model override, per-request context)
320
+ - the sub-agent orchestration (architect / designer / dba / researcher / programmer)
321
+ - the CWD-relative catalog (types / fns / vars directories)
322
+ - the REPL and one-shot CLI entry point
323
+
324
+ If you want to embed the same capabilities in your own application
325
+ rather than use a CLI, use `@aeye/gin` directly — everything ginny
326
+ does is a thin layer of tool definitions over gin's public API.
327
+
328
+ ## License
329
+
330
+ GPL-3.0
@@ -0,0 +1,193 @@
1
+ # @aeye/ginny — Program Globals, File Shapes & Gotchas
2
+
3
+ Part of the [@aeye/ginny LLM reference](./aeye-ginny.md).
4
+
5
+ ## Programmatic use (read this first)
6
+
7
+ `@aeye/ginny` is a **CLI application, not a consumable library**. Its
8
+ package `main`/`bin` both point at `dist/index.js`, whose top-level code
9
+ runs `main()` (the REPL / one-shot loop). Importing it executes the CLI;
10
+ it does not export a stable API surface for embedding.
11
+
12
+ The source module `src/ai.ts` does export internal bindings (`ai`,
13
+ `registry`, `engine`, `store`, `features`, `aiInfo`) but these are
14
+ implementation details of the CLI, not a published API. **To build your
15
+ own application with the same capabilities, depend on `@aeye/gin`
16
+ directly** (the engine, type system, and schema generation) plus
17
+ `@aeye/ai` for provider/model orchestration — ginny itself is a thin
18
+ layer of tool and prompt definitions over those.
19
+
20
+ ## Built-in program globals
21
+
22
+ Generated gin programs always have these in scope. The four natives live
23
+ under the `fns.*` namespace (wired in `src/ai.ts`, implemented in
24
+ `src/natives/*`); saved user functions and vars live at the **top level**.
25
+
26
+ ### `fns.fetch` — typed HTTP
27
+
28
+ ```
29
+ fns.fetch<R: any>({
30
+ url: text,
31
+ method?: text,
32
+ headers?: map/obj,
33
+ body?: any,
34
+ convert?: "markdown" | "raw", // default "markdown"
35
+ output: typ<R> // REQUIRED
36
+ }): R
37
+ ```
38
+
39
+ - `output` is **required** so the generic `R` is always bound at the call
40
+ site (an unbound `R` trips `define.var.type-mismatch`).
41
+ - `output: typ<text>` — content mode. `convert: "markdown"` (default)
42
+ renders HTML (with headless-browser SPA support), PDF, DOCX, XLSX as
43
+ markdown and wraps JSON/CSV/source in fenced text. `convert: "raw"`
44
+ returns the literal response body. This already gives you readable text
45
+ — there is no need to write helpers that strip HTML or parse PDFs.
46
+ - `output: typ<<obj/list>>` — JSON-API mode. The body is JSON-parsed and
47
+ type-parsed against the type; `convert` is ignored.
48
+
49
+ ### `fns.llm` — LLM with typed output
50
+
51
+ ```
52
+ fns.llm<R: any>({ prompt: text, tools?: list<any>, output?: typ<R> }): R
53
+ ```
54
+
55
+ - Omit `output` (or pass `text`) for a plain-string reply.
56
+ - Pass an `obj` type to use the structured-output channel. Other types
57
+ (enum/num/bool/list/tuple) are auto-wrapped as `{ value }` over the wire
58
+ and unwrapped before parse, so callers see the inner value.
59
+ - Bind `R` explicitly via a call-site `generic: { R: ... }` when you want
60
+ the return to read as a specific named type.
61
+ - Uses the `GIN_LLM_MODEL` (then `GIN_MODEL`) model selection.
62
+
63
+ ### `fns.log` — runtime narration
64
+
65
+ ```
66
+ fns.log({ message: any }): void
67
+ ```
68
+
69
+ Prints a runtime message to the user (stderr). For progress narration,
70
+ intermediate values, or debug breadcrumbs. Distinct from the program's
71
+ return value.
72
+
73
+ ### `fns.ask` — interactive user prompt
74
+
75
+ ```
76
+ fns.ask<R: any>({ title: text, details: text, output?: typ<R> }): optional<R>
77
+ ```
78
+
79
+ Pauses the program and prompts the human. With `output` set, the
80
+ interactive **consumer** (`src/consumer.ts`) walks the user through any
81
+ shape — obj fields, list items (add-another loop), map entries, tuples,
82
+ enum/or choices, optionals/nullables — re-prompting on parse errors.
83
+ **Each (sub)type's `docs` field becomes the user-facing label** (a prop's
84
+ `docs` wins over the type's `docs` for that field). Returns `null`
85
+ (`optional<R>`) if the user cancels, so a program must handle that branch.
86
+
87
+ ### `vars.*` and saved functions
88
+
89
+ - `vars.<name>` reads a persisted typed value; assigning via a `set`
90
+ expression writes it back (and the write is persisted after a
91
+ successful `test`).
92
+ - Every fn saved via `finish({ saveAs })` (or created by the designer) is
93
+ registered as a **top-level global** under its bare name — call it as
94
+ `<name>({ ...args })`, **not** `fns.<name>`. The `fns` namespace holds
95
+ only the four built-in natives above.
96
+
97
+ ## On-disk file shapes
98
+
99
+ All three catalogs are CWD-relative. The filename (without `.json`) is the
100
+ entry's name.
101
+
102
+ ### `./vars/<name>.json` — a var
103
+
104
+ A `{ type, value, docs }` triple:
105
+
106
+ ```json
107
+ {
108
+ "type": { "name": "text", "options": { "pattern": "^https?://" } },
109
+ "value": "https://api.example.com",
110
+ "docs": "production API root"
111
+ }
112
+ ```
113
+
114
+ ### `./types/<Name>.json` — a type
115
+
116
+ A gin `TypeDef` (serialized type descriptor). For example a `Task` obj
117
+ type extends `obj` with props. Inspect any saved type with `get_type`
118
+ (architect) — it renders the full `toCodeDefinition()`.
119
+
120
+ ### `./fns/<name>.json` — a function
121
+
122
+ A `function`-typed `TypeDef` whose **body lives at `call.get`** — gin's
123
+ native callable shape. The path walker dispatches invocation, argument
124
+ binding, and `recurse` with no ginny-side wrapping. The top-level `docs`
125
+ field is the function's description (what `search_fns` surfaces). When the
126
+ designer declared `call.types` aliases, they are preserved verbatim so the
127
+ saved fn keeps its compact form.
128
+
129
+ ```
130
+ {
131
+ "name": "fn",
132
+ "docs": "compute the factorial of n",
133
+ "call": {
134
+ "args": { "name": "obj", "props": { "n": { "type": { "name": "num" } } } },
135
+ "returns": { "name": "num" },
136
+ "get": { /* the ExprDef body */ }
137
+ }
138
+ }
139
+ ```
140
+
141
+ ## Worked examples (REPL transcripts)
142
+
143
+ ```
144
+ > add 2 and 3
145
+ → simple mode: write → test (SUCCESS: 5) → finish. 5
146
+
147
+ > compute the factorial of 6
148
+ → find_or_create_functions("factorial") → designer spawns a programmer
149
+ that writes the recursive body → test (720) → finish(saveAs). 720
150
+
151
+ > fetch the title of example.com
152
+ → fns.fetch(url, output: typ<text>) returns markdown → extract title.
153
+
154
+ > remember my api base url is https://api.example.com as 'apiBaseUrl'
155
+ → dba creates ./vars/apiBaseUrl.json.
156
+
157
+ > print my api base url
158
+ → program reads vars.apiBaseUrl.
159
+
160
+ > define a Task type with title, done, due
161
+ → architect creates ./types/Task.json (obj extension with props).
162
+
163
+ > add an `assignee` field to Task (optional)
164
+ → edit_type — backwards-compatible widening accepted.
165
+ ```
166
+
167
+ ## Gotchas
168
+
169
+ - **`output` is mandatory on `fns.fetch`.** Omitting it leaves `R`
170
+ unbound and downstream type checks fail.
171
+ - **Saved fns are top-level, not under `fns.*`.** Call `summarizePage({…})`,
172
+ not `fns.summarizePage({…})`.
173
+ - **Catalog search is keyword matching, not semantic/embedding search.**
174
+ Below `GIN_SEARCH_THRESHOLD` (20) entries it returns everything; above,
175
+ it counts literal keyword occurrences. Pick keywords that actually
176
+ appear in names/docs.
177
+ - **`finish` can reject a save** even after a passing test: too many
178
+ warnings (`GIN_MAX_WARNINGS`, default 5), any errors, or complexity over
179
+ `GIN_MAX_COMPLEXITY` (default 400). The fix is decomposition into helper
180
+ functions.
181
+ - **Edits are one-directional.** `edit_type`/`edit_fn` accept widening
182
+ (and narrowing returns for fns) but reject breaking changes; create a
183
+ new name instead.
184
+ - **The deepest programmer (depth 2 of 3) cannot delegate** — it must
185
+ write any needed function inline. `MAX_PROGRAMMER_DEPTH = 3`.
186
+ - **Web tools require Tavily.** Without `TAVILY_API_KEY`, `research`,
187
+ `web_search`, and `web_get_page` are not exposed. (Generated programs
188
+ can still fetch + convert content directly via `fns.fetch`.)
189
+ - **First run exits without doing work** — it only scaffolds `config.json`
190
+ and `.gitignore`. Re-run after adding credentials.
191
+ - **`fns.ask` returns `optional<R>`** — programs must handle the cancel
192
+ (`null`) branch; put `docs` on every field of the output type or the
193
+ user sees raw field names instead of labels.