aiex-cli 0.0.1-beta.9 → 0.0.2-beta.1

package/README.md

@@ -21,10 +21,9 @@ npm install -g aiex-cli
 ```
 
 ```bash
-aiex schema --init                      # set up .aiex/schema/ directory
+aiex web                                # configure schemas, AI settings, and run extractions in the browser
 aiex schema                             # generate SQLite from JSON Schema files
-aiex extract -s invoice -f invoice.pdf  # extract data with AI
-aiex extract -s invoice -f invoice.pdf --db  # extract and insert into database
+aiex extract -s invoice -f invoice.pdf  # extract data with AI and insert into database
 ```
 
 <br>
@@ -33,32 +32,26 @@ aiex extract -s invoice -f invoice.pdf --db  # extract and insert into database
 
 - **JSON Schema → SQLite** — Define tables as JSON Schema files, generate Drizzle ORM schema, and migrate to SQLite
 - **Visual Editor** — Browser-based UI for designing schemas without writing JSON by hand
+- **Web Extraction Workbench** — Run text/file extraction in the browser, review audit history, inspect details, retry failed runs, and delete history records
 - **AI Extraction** — Extract structured data from text, images, and PDFs using any OpenAI-compatible provider (OpenAI, Anthropic, Ollama, DeepSeek, local models, etc.)
+- **Interactive Mode** — Run `aiex extract` without arguments for a guided extraction workflow
+- **Batch Mode** — `aiex extract -d <dir>` processes entire directories with optional glob filtering
+- **Extraction Audit Trail** — Every CLI/Web extraction is recorded with status, input source, output path, token usage, database inserts, and errors
 - **Built-in Model Registry** — Knows capabilities of 2000+ models (vision, structured output) so you don't have to guess
 
 <br>
 
 ## 🚀 Getting Started
 
-### 1. Initialize
-
-```bash
-aiex schema --init
-```
-
-Creates a `.aiex/` directory with example schemas to get you started.
-
-Add your own JSON Schema files to `.aiex/schema/` (one file per table), then run `aiex schema` to migrate them into the database.
-
-### 2. Visual Editor
+### 1. Configure In Web UI
 
 ```bash
 aiex web
 ```
 
-Opens a browser UI where you can visually design and manage your schemas, configure AI settings, preview extraction prompts, and apply changes to the database.
+Opens a browser UI where you can visually design and manage your schemas, configure AI settings, run text/file extraction, inspect extraction history, retry failed runs, preview extraction prompts, browse inserted data, and apply changes to the database.
 
-### 3. Generate Database
+### 2. Generate Database
 
 ```bash
 aiex schema
@@ -66,39 +59,59 @@ aiex schema
 
 Converts your JSON Schema files into a SQLite database with full migration support.
 
-### 4. Extract Data
+### 3. Extract Data
 
 ```bash
-aiex extract -s <schema> -f <file>     # from file (txt, pdf, png, jpg, ...)
-aiex extract -s <schema> -t <text>     # from text
-aiex extract -s <schema> -f <file> -m <model>  # specify AI model (overrides auto-selection)
+aiex extract                              # interactive mode (prompts for schema & input)
+aiex extract -s <schema> -f <file>        # from file (txt, pdf, png, jpg, ...)
+aiex extract -s <schema> -t <text>        # from text
+aiex extract -s <schema> -f <file> -m <model>     # specify AI model (overrides auto-selection)
+aiex extract -s <schema> -f <file> --no-insert    # extract and save JSON without inserting into SQLite
+aiex extract -s <schema> -d <directory>           # batch extract all supported files in a directory
+aiex extract -s <schema> -d <dir> -g "*.pdf"      # batch with glob filter
+aiex extract history                              # list extraction audit records
+aiex extract show <audit-id>                      # show full audit record JSON
+aiex extract retry <audit-id>                     # retry a previous extraction
+aiex extract rm <audit-id>                        # delete an audit record and cached upload
 ```
 
 The AI reads your document and outputs structured JSON matching your schema.
 
 **Examples:**
 ```bash
-aiex extract -s paper -f research.pdf              # save result to .aiex/extracted/
-aiex extract -s paper -f research.pdf --db         # also insert into SQLite database
+aiex extract                                      # interactive mode
+aiex extract -s paper -f research.pdf              # save result to .aiex/extracted/ and insert into database
+aiex extract -s paper -f research.pdf --no-insert   # save result only, skip database insert
 aiex extract -s paper -f research.pdf -m gpt-4o    # use a specific model
+aiex extract -s paper -d ./papers -g "*.pdf"        # batch extract PDFs from a directory
+aiex extract history                                # inspect recent extraction runs
 ```
-Saves the extracted result to `.aiex/extracted/<schema-name>-<timestamp>.json` with fields like `title`, `firstAuthor`, `journal`, `year` — exactly as defined in your schema. Add `--db` to also insert the data directly into the SQLite database.
+Saves the extracted result to `.aiex/extracted/<schema-name>-<timestamp>.json` with fields like `title`, `firstAuthor`, `journal`, `year` — exactly as defined in your schema. Data is automatically inserted into the SQLite database.
 
 By default, aiex automatically selects a model based on your input type (vision-capable for images, structured output for text). Use `--model` / `-m` to override and specify any model from your AI configuration.
 
+Every extraction is also recorded under `.aiex/extracted/_audit/`. Audit records include the run status (`running`, `succeeded`, `failed`, or `stale`), schema name, input source, output file, token usage, inserted table rows, retry lineage, and error message. Uploaded files used by Web extraction are cached under `.aiex/uploads/` so failed or historical file extractions can be retried. Deleting an audit record removes its cached upload, but keeps extracted JSON result files to avoid accidental data loss.
+
 <br>
 
 ## 📖 Commands
 
 | Command | Description |
 | --- | --- |
-| `aiex schema --init` | Scaffold `.aiex/` directory with example schemas |
 | `aiex schema` | Parse JSON Schema files and migrate to SQLite |
 | `aiex schema --generate` | Generate Drizzle schema code only (skip migration) |
 | `aiex web` | Launch visual schema editor in browser |
-| `aiex extract -s <name> -f <file>` | Extract structured data from documents via AI |
-| `aiex extract -s <name> -f <file> --db` | Extract and insert into SQLite database |
+| `aiex extract` | Interactive mode — prompts for schema and input source |
+| `aiex extract -s <name> -f <file>` | Extract structured data from documents and insert into SQLite database |
 | `aiex extract -s <name> -f <file> -m <model>` | Extract with a specific AI model |
+| `aiex extract -s <name> -f <file> --no-insert` | Extract and save JSON without inserting into SQLite |
+| `aiex extract -s <name> -d <dir>` | Batch extract all supported files in a directory |
+| `aiex extract -s <name> -d <dir> -g "*.pdf"` | Batch extract with glob filter |
+| `aiex extract history` | List extraction audit records |
+| `aiex extract show <audit-id>` | Show a full extraction audit record |
+| `aiex extract retry <audit-id>` | Retry a previous extraction run |
+| `aiex extract retry <audit-id> --no-insert` | Retry without inserting into SQLite |
+| `aiex extract rm <audit-id>` | Delete an audit record and its cached upload |
 | `aiex doctor` | System and configuration diagnostics |
 | `aiex completion bash\|zsh\|fish` | Generate shell completion scripts |
 
@@ -133,6 +146,17 @@ aiex works with any OpenAI-compatible API provider. Configure in the Web UI (AI
 
 The built-in model registry automatically suggests capabilities for 2000+ models from providers including OpenAI, Anthropic, Google, Meta, Mistral, DeepSeek, Alibaba Cloud, and more.
 
+### Langfuse Tracing
+
+aiex can send AI model interaction traces to [Langfuse](https://langfuse.com) for monitoring and debugging.
+
+- **Enable** — In Web UI → AI Settings → Langfuse Tracing, toggle on and enter your Langfuse Secret Key / Public Key
+- **Self-hosted** — Optionally set a custom Host URL; defaults to `https://us.cloud.langfuse.com`
+- **No impact when disabled** — No tracing is sent if keys are left empty
+- **Non-blocking** — Misconfigured keys will not affect extraction
+
+Once enabled, every `aiex extract` call is automatically traced with full request/response payloads, token usage, and latency.
+
 <br>
 
 ## 🙏 Acknowledgments