ei-tui 0.1.3 → 0.1.4

package/README.md

@@ -20,7 +20,7 @@ There's no other usage, debugging, analytics, tracking, or history information s
 
 If there's a problem with the system, you need to tell me here on GitHub, or on Bluesky, or Discord, or whatever. There's no "report a bug" button, no "DONATE" link in the app.
 
-Don't get me wrong -I absolutely want to fix whatever problem you run into, or hear about the feature you want - but your Ei system, and the data you build with it, is yours.
+Don't get me wrong - I absolutely want to fix whatever problem you run into, or hear about the feature you want - but your Ei system, and the data you build with it, is yours.
 
 That's what "Local First" means.
 
@@ -36,15 +36,14 @@ That I can't decrypt.
 
 Even if I wanted to (I definitely do not), I wouldn't be able to divulge your information because **You** are the only one that can generate the key. It's not a public/private keypair, it's not a "handshake".
 
-It's your data - I have no right to it, and neither does anyone else except you.
+It's *your* data - I have no right to it, and neither does anyone else except you.
 
 ## What's a Persona?
 
 At the core of the technology, LLM "Agents" are made up of two or three components, depending on who you ask:
 
 1. System Prompt
-2. User Prompt
-    a. Which can be broken into "Messages", but they're still basically the User Prompt
+2. User Prompt (which can be broken into "Messages", but they're still basically the User Prompt)
 
 The "System Prompt" is the part where you usually say
 
@@ -55,7 +54,7 @@ The "User Prompt" is the part where you put your messages
 > user: "OMG ARE YOU REALLY A PIRATE?!"
 > assistant: "Yar."
 
-A "Persona" is the combination of these two pieces of data, plus some _personality_. The reason I didn't call it an "Agent" is because Personas aren't static<sup>1</sup> - they'll grow and adapt as you talk to them. See the [Core Readme](core/README.md) for more information!
+A "Persona" is the combination of these two pieces of data, plus some _personality_. The reason I didn't call it an "Agent" is because Personas aren't static<sup>1</sup> - they'll grow and adapt as you talk to them. See the [Core Readme](src/README.md) for more information!
 
 > <sup>1</sup>: By default. You can make them static.
 
@@ -83,7 +82,7 @@ Optionally, users can opt into a server-side data sync. This is ideal for users
 
 ### Web
 
-When you access Ei via https://ei.flare576.com, your browser will download the assets and walk you through onboarding. If you're running a Local LLM on port :1234 it will auto-detect it, otherwise it will  allowing you to enter it.
+When you access Ei via https://ei.flare576.com, your browser will download the assets and walk you through onboarding. If you're running a Local LLM on port :1234 it will auto-detect it, otherwise it prompts you to enter one.
 
 Then you'll land on the chat interface. As you enter messages, they'll go to *YOUR* server. As Ei discovers information about you, summaries will be built with *YOUR* server, and data will be stored to *YOUR* LocalStorage in *YOUR* browser.
 
@@ -107,43 +106,28 @@ More information (including commands) can be found in the [TUI Readme](tui/READM
 
 ### Opencode
 
-Opencode saves all of its sessions locally, either in a JSON structure or, if you're running the latest version, in a SQLite DB. If you enable the integration, Ei will pull all of the conversational parts of those sessions and summarize them, pulling out details, quotes, and keeping the summaries up-to-date.
+Ei gives OpenCode a persistent memory. Yes, this is a dynamic, perpetual RAG — I didn't plan it that way, but here we are.
 
-Then, Opencode can call into Ei and pull those details back out.
+Opencode saves all of its sessions locally, either in a JSON structure or, if you're running the latest version, in a SQLite DB. If you enable the integration, Ei will pull all of the conversational parts of those sessions and summarize them, pulling out details, quotes, and keeping the summaries up-to-date.
 
-Yes, I did make a dynamic, perpetual RAG. No, I didn't do it on purpose; that's why you always have a side-project or two going. See [TUI Readme](tui/README.md)
+Then, Opencode can call into Ei and pull those details back out. That's why you always have a side-project or two going. See [TUI Readme](tui/README.md)
 
 ## Technical Details
 
 This project is separated into five (5) logical parts:
 
-1. Ei Core
-    a. Location: `/src`
-    b. Purpose: Shared between TUI and Web, it's The event-driven core of Ei, housing:
-        i. Business logic
-        ii. Prompts
-        iii. Integrations
-2. Ei Online
-    a. Location: `/web`
-    b. Purpose: Provides a web interface for Ei.
-    c. Deployed to: https://ei.flare576.com
-3. Ei Terminal User Interface (TUI)
-    a. Location: `/tui`
-    b. Purpose: Provides a TUI interface for Ei
-    c. Deployed to: NPM for you to install
-4. Ei API
-    a. Location: `/api`
-    b. Purpose: Provides remote sync for Ei.
-    c. Deployed to: https://ei.flare576.com/api
-5. Ei Command Line Interface (CLI)
-    a. Location: `/src/cli`
-    b. Purpose: Provides a CLI interface for Opencode to use as a tool
-    c. Technically, ships with the TUI
+| Part | Location | Purpose | Deployed To |
+|------|----------|---------|-------------|
+| Ei Core | `/src` | Shared between TUI and Web. The event-driven core of Ei, housing business logic, prompts, and integrations. | (library) |
+| Ei Online | `/web` | Web interface for Ei. | https://ei.flare576.com |
+| Ei Terminal UI (TUI) | `/tui` | TUI interface for Ei. | NPM for you to install |
+| Ei API | `/api` | Remote sync for Ei. | https://ei.flare576.com/api |
+| Ei CLI | `/src/cli` | CLI interface for Opencode to use as a tool. Technically ships with the TUI. | (ships with TUI) |
 
 ## Requirements
- [Bun](https://bun.sh) runtime (>=1.0.0)
- Local LLM provider (LM Studio, Ollama, etc.)
-    * OR API access to a remote LLM host (Anthropic, OpenAI, Bedrock, your uncle's LLM farm, etc.)
+
+- [Bun](https://bun.sh) runtime (>=1.0.0)
+- A local LLM (LM Studio, Ollama, etc.) OR API access to a cloud provider (Anthropic, OpenAI, Bedrock, your uncle's LLM farm, etc.)
 
 ## LM Studio Setup
 
@@ -165,6 +149,19 @@ npm run build    # Compile TypeScript
 npm run test     # Run tests
 ```
 
+## Releases
+
+Tag a version to publish automatically:
+
+```bash
+# bump version in package.json
+git commit -am "chore: bump to v0.1.4"
+git tag v0.1.4
+git push && git push --tags
+```
+
+GitHub Actions picks up the tag and publishes to npm with provenance via OIDC. No stored secrets.
+
 ## Project Structure
 
 See `AGENTS.md` for detailed architecture and contribution guidelines.

package/package.json

@@ -1,7 +1,11 @@
 {
   "name": "ei-tui",
-  "version": "0.1.3",
+  "version": "0.1.4",
   "author": "Flare576",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/flare576/ei.git"
+  },
   "engines": {
     "bun": ">=1.0.0"
   },

package/src/README.md

@@ -22,8 +22,12 @@ As the user uses the system, it tries to keep track of several data points for t
         + 0.0: The user never EVER wants to talk about or hear about the subject
         + 1.0: Every message to and from the user should be about this person, place, or thing
     * Current: How much the user has talked or heard about a subject, where:
-        + 0.0: Obi-Wan Kenobi ...now that’s a name I’ve not heard in a long time
+        + 0.0: Obi-Wan Kenobi ...now that's a name I've not heard in a long time
         + 1.0: The user just spent 4 hours talking about Star Wars
+- Strength: The system will try to gauge how strongly you exhibit a Trait
+    * 1.0 on "Visual Learner" would mean that you've said or shown that it is the absolute best way for you to learn
+    * 0.0 on "Public Speaker" would mean you've said or shown that you have no desire, aptitude, or willingness to present
+- Validated: "Facts" have proven almost as hard to get right as Traits, so I added a way for Ei and you to mark the ones that are true as "Validated"
 
 Each of those types represents a piece of what the system "knows" about the person, and all but "Traits" are kept up-to-date as the person chats with Personas, but not on always on every message. On each message to a Persona, a check is made:
 
@@ -94,3 +98,83 @@ This is largely tracked by exposure, but expiration is dictated by an Agent.
 
 After we've removed irrelevant topics, this is the Agent's opportunity to add NEW topics that might be of interest to the Persona (and the user). Again, it's a prompt to an agent if the Persona doesn't have its full capacity of Topics.
 
+# Opencode Importer
+
+The current implementation of the importer is very, very simple. You could probably just read the code to get the idea, but, essentially:
+
+1. If there's any queue, skip
+2. Look at the `last_extraction_ts` and find the OpenCode Session with the last_updated time closest to it
+3. Check if we've already imported this session, and what the cutoff is for "already seen" messages
+4. Wipe the Personas history and mark them as archived
+5. Write the messages to the Persona, marking old messages as `[p,r,o,f]` processed
+6. Queue the extract
+7. Bump the `last_extraction_ts` to that session's last_updated timestamp
+
+That's it. We slowly process through OpenCode's backlog, chronologically, until we finally catch up. Once we do, we just parse each message/exchange and extract for now. I think eventually I'll add a gate that says "if last_updated is less than 24h, skip this extraction" so that each session has a better chance of being "Complete" and offering the extraction process a whole look.
+
+## Failed Approaches
+
+In the spirit of "Right down what you tried, so you don't try it again," here are the other ways we've attempted to pull in the data.
+
+# V1 - Greedy
+
+The first mechanism used a three-phase process. First, we'd pull **every** message from OpenCode into Ei, assigning them to personas. This was before we had markers on each message to indicate if they'd been processed for `[Facts, tRaits, People, tOpics]` (the `[f,r,p,o]`, but I usually switch the order to `[p,r,o,f]` because I can't remember the other order). On subsequent runs, we'd just tack the latest messages onto the end of the conversation.
+
+Second, we'd queue up Topic updates and quote retrieval starting with the most recent messages. The goal was to get the latest, hottest quotes out of the system as soon as possible so the user gets "immediate value"
+
+Last, we'd start queuing up Fact, Trait, and People updates from the oldest records.
+
+## Why Didn't This Work
+
+You ever try to tell an LLM a story by starting at the end and working your way backward? Don't bother, it doesn't work. Trying to process the last messages first was an attempt to get visible value as soon as possible, but that value was a lie.
+
+Oh, and that initial queue for quotes was about 450 items.
+
+# V2 - Clever
+
+The second approach ended up being roughly 800 lines of "Clever" code. I call it that because Opus found at least 4 edge cases in the logic that we added dials and trackers around, so this quote applied:
+
+> Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?
+> -- Brian Kernighan, 1974
+
+The approach broke the messages into a "recent" timeframe and the rest. We had just added message roll-off to the main personas with a rule set of "Always keep at least 200 messages, but after that roll off any message older than 14 days," so we used the same approach for Opencode Agent personas.
+
+We loaded the last 14 days of messages... Which was tricky because we _also_ wanted to keep messages tied to their "sessions" for context, so if a user added a message to a session from last year, we'd need to pull old messages (that we already processed) and the new messages (that we hadn't) into a block that the system could parse correctly.
+
+Oh, and the "older than 14 days" messages we had to process in, too, so we needed _another_ timestamp tracker for that...
+
+And sometimes that process will _also_ be split in the same way as the recent messages...
+
+Aaaand the most recent 14 days of messages, queued for all four data types, resulted in an initial queue of 300 items **scans**, and once each of those scans found 10 Topics to talk about, the queue jumped to 3,000 instantly.
+
+# The Processing Loop
+
+The processor runs a tight loop every 100ms. On each tick it checks: is the queue idle? Is there a request waiting? If yes, it dequeues the highest-priority item and hands it to the LLM. One at a time. Always.
+
+This is intentional. Concurrent LLM calls sound appealing until you're watching a persona give contradictory answers because two extractions ran in parallel on the same data. Boring serialization beats exciting race conditions.
+
+# Context Windows
+
+Personas don't send their entire message history to the LLM. By default, only messages from the last 8 hours are included (`context_window_hours`, configurable per persona). Older messages are still stored — they're just not in the prompt.
+
+Message rolloff works differently: messages are kept until there are at least 200 of them _and_ any are older than 14 days. So a persona you chat with daily will roll off old messages gradually; one you chat with twice a year will keep everything.
+
+# Embeddings
+
+Every fact, trait, person, topic, and quote gets a vector embedding when it's created or updated. The model is `all-MiniLM-L6-v2` (384 dimensions) — small enough to run locally, accurate enough to be useful.
+
+When building a system prompt for a response, the system doesn't just dump all your data into the context. It uses cosine similarity against the current message to find the most relevant items — up to 15 per type, with a 0.3 similarity threshold. Everything below the threshold gets left out.
+
+The model runs via `fastembed` in Bun/Node and `@huggingface/transformers` in the browser. Both are loaded lazily so the bundler doesn't have a bad day.
+
+# Encryption
+
+The sync feature uses AES-GCM-256 with a key derived via PBKDF2 (310,000 iterations) from `username:passphrase`. The key never leaves your device. The server receives an encrypted blob it can't read.
+
+There's a subtle trick for the user ID: to identify your data on the server without sending credentials, the system encrypts a fixed known plaintext (`"the_answer_is_42"`) with a fixed IV using your key. Same credentials always produce the same ciphertext, which becomes your server-side ID. No account, no lookup table — your identity _is_ your credentials.
+
+# Heartbeats
+
+Each persona has a heartbeat timer. If a persona hasn't had activity for 30 minutes (configurable via `heartbeat_delay_ms`), the system queues a check-in prompt. The persona "wakes up," considers what's been going on, and may have something to say.
+
+Whether they do is up to them and the prompt. Some personas are chatty. Some are not.

package/src/cli/README.md

@@ -1,10 +1,7 @@
 # The CLI
-
-It's actually super straight-forward
-
 ```sh
 ei                             # Start the TUI
-ei "query string"              # Return up to 10 "query string" across all types
+ei "query string"              # Return up to 10 results across all types
 ei -n 5 "query string"         # Return up to 5 results
 ei facts -n 5 "query string"   # Return up to 5 facts
 ei traits -n 5 "query string"  # Return up to 5 traits
@@ -13,10 +10,12 @@ ei topics -n 5 "query string"  # Return up to 5 topics
 ei quotes -n 5 "query string"  # Return up to 5 quotes
 ei --id <id>                   # Look up a specific entity by ID
 echo <id> | ei --id            # Look up entity by ID from stdin
+ei --install                   # Install the Ei tool for OpenCode
 ```
 
-# An Agentic Tool
+Type aliases: `fact`, `trait`, `person`, `topic`, `quote` all work (singular or plural).
 
+# An Agentic Tool
 
 The `--id` flag is designed for piping. For example, search for a topic and then fetch the full entity:
 
@@ -24,24 +23,33 @@ The `--id` flag is designed for piping. For example, search for a topic and then
 ei "memory leak" | jq '.[0].id' | ei --id
 ```
 
-To register Ei as an explicit OpenCode tool (optional — agents can also just call `ei` via shell):
+# OpenCode Integration
 
-```bash
-mkdir -p ~/.config/opencode/tools
+## Quick Install
+
+```sh
+ei --install
 ```
 
-Create `~/.config/opencode/tools/ei.ts`:
+This writes `~/.config/opencode/tools/ei.ts` with a complete tool definition. Restart OpenCode to activate.
 
-```typescript
-import { tool } from "@opencode-ai/plugin"
+## What the Tool Provides
 
-export default tool({
-  description: "Search the user's Ei knowledge base for facts, people, topics, traits, and quotes",
-  args: {
-    query: tool.schema.string().describe("Search query"),
-  },
-  async execute(args) {
-    return await Bun.$`ei ${args.query}`.text()
-  },
-})
-```
+The installed tool gives OpenCode agents access to all five data types with proper Zod-validated args:
+
+| Arg | Type | Description |
+|-----|------|-------------|
+| `query` | string (required) | Search text, or entity ID when `lookup=true` |
+| `type` | enum (optional) | `facts` \| `traits` \| `people` \| `topics` \| `quotes` — omit for balanced results |
+| `limit` | number (optional) | Max results, default 10 |
+| `lookup` | boolean (optional) | If true, fetch single entity by ID |
+
+## Output Shapes
+
+All search commands return arrays. Each result includes a `type` field.
+
+**Fact / Trait / Person / Topic**: `{ type, id, name, description, sentiment, ...type-specific fields }`
+
+**Quote**: `{ type, id, text, speaker, timestamp, linked_items[] }`
+
+**ID lookup** (`lookup: true`): single object (not an array) with the same shape.

package/src/cli/retrieval.ts

@@ -1,4 +1,5 @@
 import type { StorageState, Quote, Fact, Trait, Person, Topic } from "../core/types";
+import { crossFind } from "../core/utils/index.ts";
 import { join } from "path";
 import { readFile } from "fs/promises";
 import { getEmbeddingService, findTopK } from "../core/embedding-service";
@@ -249,21 +250,8 @@ export async function lookupById(id: string): Promise<({ type: string } & Record
     return null;
   }
 
-  const collections: Array<{ type: string; source: Array<{ id: string; [k: string]: unknown }> }> = [
-    { type: "fact", source: state.human.facts },
-    { type: "trait", source: state.human.traits },
-    { type: "person", source: state.human.people },
-    { type: "topic", source: state.human.topics },
-    { type: "quote", source: state.human.quotes },
-  ];
-
-  for (const { type, source } of collections) {
-    const entity = source.find(item => item.id === id);
-    if (entity) {
-      const { embedding, ...rest } = entity;
-      return { type, ...rest };
-    }
-  }
-
-  return null;
+  const found = crossFind(id, state.human);
+  if (!found) return null;
+  const { type, embedding, ...rest } = found;
+  return { type, ...rest };
 }

package/src/cli.ts

@@ -12,6 +12,7 @@
  */
 
 import { parseArgs } from "util";
+import { join } from "path";
 import { retrieveBalanced, lookupById } from "./cli/retrieval";
 
 const TYPE_ALIASES: Record<string, string> = {
@@ -50,6 +51,7 @@ Types:
 Options:
   --number, -n     Maximum number of results (default: 10)
   --id             Look up entity by ID (accepts value or stdin)
+  --install        Write the Ei tool file to ~/.config/opencode/tools/
   --help, -h       Show this help message
 
 Examples:
@@ -62,6 +64,68 @@ Examples:
 `);
 }
 
+function buildOpenCodeToolContent(): string {
+  const lines = [
+    'import { tool } from "@opencode-ai/plugin"',
+    '',
+    'export default tool({',
+    '  description: [',
+    '    "Search the user\'s Ei knowledge base \u2014 a persistent memory store built from conversations.",',
+    '    "Returns facts, personality traits, people, topics of interest, and quotes.",',
+    '    "Use this to recall anything about the user: preferences, relationships, or past discussions.",',
+    '    "Results include entity IDs that can be passed back with lookup=true to get full detail.",',
+    '  ].join(" "),',
+    '  args: {',
+    '    query: tool.schema.string().describe(',
+    '      "Search text, or an entity ID when lookup=true. Supports natural language."',
+    '    ),',
+    '    type: tool.schema',
+    '      .enum(["facts", "traits", "people", "topics", "quotes"])',
+    '      .optional()',
+    '      .describe(',
+    '        "Filter to a specific data type. Omit to search all types (balanced across all 5)."',
+    '      ),',
+    '    limit: tool.schema',
+    '      .number()',
+    '      .int()',
+    '      .positive()',
+    '      .default(10)',
+    '      .optional()',
+    '      .describe("Maximum number of results to return. Default: 10."),',
+    '    lookup: tool.schema',
+    '      .boolean()',
+    '      .optional()',
+    '      .describe(',
+    '        "If true, treat query as an entity ID and return that single entity in full detail."',
+    '      ),',
+    '  },',
+    '  async execute(args) {',
+    '    const cmd: string[] = ["ei"];',
+    '    if (args.lookup) {',
+    '      cmd.push("--id", args.query);',
+    '    } else {',
+    '      if (args.type) cmd.push(args.type);',
+    '      if (args.limit && args.limit !== 10) cmd.push("-n", String(args.limit));',
+    '      cmd.push(args.query);',
+    '    }',
+    '    return Bun.$`${cmd}`.text();',
+    '  },',
+    '})',
+    '',
+  ];
+  return lines.join('\n');
+}
+
+async function installOpenCodeTool(): Promise<void> {
+  const toolsDir = join(process.env.HOME || "~", ".config", "opencode", "tools");
+  const toolPath = join(toolsDir, "ei.ts");
+
+  await Bun.$`mkdir -p ${toolsDir}`;
+  await Bun.write(toolPath, buildOpenCodeToolContent());
+  console.log(`✓ Installed Ei tool to ${toolPath}`);
+  console.log(`  Restart OpenCode to activate.`);
+}
+
 async function main(): Promise<void> {
   const args = process.argv.slice(2);
 
@@ -82,6 +146,11 @@ async function main(): Promise<void> {
     process.exit(0);
   }
 
+  if (args[0] === "--install") {
+    await installOpenCodeTool();
+    process.exit(0);
+  }
+
 
   // Handle --id flag: look up entity by ID
   const idFlagIndex = args.indexOf("--id");