cli-mem 0.3.6__tar.gz → 0.4.0__tar.gz

{cli_mem-0.3.6 → cli_mem-0.4.0}/ARCHITECTURE.md

@@ -17,12 +17,15 @@ Technical overview of how mem works under the hood.
 ┌─────────────────────────────────────────────────────────┐
 │                    src/mem/cli.py                        │
 │                                                         │
-│  mem <query>          mem sync         mem session       │
-│  mem init <shell>     mem stats        mem forget        │
-│  mem _capture (hidden)                                   │
-└───┬────────────┬──────────────┬──────────────┬──────────┘
-    │            │              │              │
-    ▼            ▼              ▼              ▼
+│  mem <query>       mem save         mem run              │
+│  mem list          mem session      mem stats            │
+│  mem forget        mem export       mem import           │
+│  mem group *       mem saved *      mem vars *           │
+│  mem init <shell>                                       │
+│  mem _capture (hidden)  mem _sync (hidden)              │
+└───┬────────────┬──────────┬──────────┬──────────────────┘
+    │            │          │          │
+    ▼            ▼          ▼          ▼
 ┌────────┐ ┌─────────┐ ┌──────────┐ ┌──────────────────┐
 │capture │ │ search  │ │ patterns │ │    storage        │
 │  .py   │ │   .py   │ │   .py    │ │      .py         │
@@ -30,19 +33,36 @@ Technical overview of how mem works under the hood.
 │ hook   │ │ score   │ │ Apple FM │ │ JSONL read/write  │
 │ capture│ │ rank    │ │ SDK      │ │ JSON read/write   │
 │ session│ │ filter  │ │ guided   │ │ rotate / forget   │
-│ tracker│ │ dedup   │ │ gen      │ │                   │
+│ tracker│ │ dedup   │ │ gen      │ │ groups / vars     │
+│ auto-  │ │         │ │ caching  │ │                   │
+│ sync   │ │         │ │          │ │                   │
 └───┬────┘ └────┬────┘ └────┬─────┘ └────────┬──────────┘
     │           │           │                 │
-    └───────────┴───────────┴─────────────────┘
+    ▼           ▼           ▼                 ▼
+┌────────┐ ┌─────────┐ ┌──────────┐ ┌──────────────────┐
+│groups  │ │variables│ │_generable│ │   models.py      │
+│  .py   │ │   .py   │ │   .py    │ │                  │
+│        │ │         │ │          │ │ Pydantic schemas  │
+│ save   │ │ parse   │ │ @fm.     │ │ CapturedCommand  │
+│ list   │ │ resolve │ │ generable│ │ WorkSession      │
+│ export │ │ detect  │ │ types    │ │ GroupFile / Group │
+│ import │ │ credent.│ │          │ │ VarsFile          │
+│ scope  │ │ store   │ │          │ │ PatternFile       │
+└────────┘ └─────────┘ └──────────┘ └──────────────────┘
                          │
                          ▼
 ┌─────────────────────────────────────────────────────────┐
-│                    ~/.mem/ (filesystem)                   │
+│                    ~/.mem/ (filesystem)                  │
 │                                                         │
-│  repos/              sessions/           patterns/       │
+│  repos/              sessions/           patterns/      │
 │    myapp.jsonl         2026-03-05.jsonl    kubectl.json  │
 │    infra.jsonl         2026-03-06.jsonl    docker.json   │
 │    _global.jsonl                           git.json      │
+│                                                         │
+│  groups/                                                │
+│    repos/                                               │
+│      myapp.json      vars.json            .sync_counter │
+│    _global.json      .session_state.json                │
 └─────────────────────────────────────────────────────────┘
 ```
 
@@ -76,6 +96,11 @@ capture.SessionTracker.update()
     │ checks idle time and repo change
     │ closes session if boundary detected
     ▼
+Auto-sync check
+    │ increments capture counter
+    │ every 20 captures: spawns `mem _sync` as detached background process
+    │   └── pattern extraction + data rotation, fully silent
+    ▼
 Done (total overhead: <5ms added to prompt)
 ```
 
@@ -112,24 +137,32 @@ Deduplicate by command string (keep highest score)
 Return top N sorted by score descending
 ```
 
-### Pattern Extraction
+### Automatic Pattern Extraction
 
 ```
-User runs: mem sync
+Every 20 captured commands:
     │
     ▼
-patterns.sync_all_patterns()
+capture.py spawns `mem _sync` as detached background process
+    │ subprocess.Popen with start_new_session=True
+    │ no stdout, no stderr — completely invisible
+    ▼
+patterns.sync_all_patterns(silent=True)
     │ reads ALL commands from all repo files
     │ groups by tool (first token)
     │ skips tools with <5 commands
     │
     ▼ (for each tool with enough data)
 patterns.extract_patterns_for_tool()
+    │
+    ├── Load cached processed_commands from existing PatternFile
+    ├── Skip already-processed commands (cache hit)
     │
     ├── If apple-fm-sdk available:
-    │   │ builds prompt with command list
-    │   │ calls Apple FM with Pydantic guided generation
-    │   └── returns PatternExtractionResult
+    │   │ generalize each NEW unique command individually
+    │   │   └── fresh LanguageModelSession per command (context safety)
+    │   │ merge with cached generalizations
+    │   └── aggregate frequencies by pattern (code)
     │
     └── If not available:
         │ groups identical commands
@@ -138,16 +171,78 @@ patterns.extract_patterns_for_tool()
     ▼
 storage.write_patterns()
     │ writes to ~/.mem/patterns/<tool>.json
+    │ includes processed_commands list for caching
     │ atomic: write tmp file, then rename
     ▼
 storage.rotate()
     │ removes commands older than 90 days
     │ deletes session files older than 30 days
-    │ NEVER touches patterns/
+    │ NEVER touches patterns/ (accumulated learning)
+    ▼
+Done (user never sees any of this)
+```
+
+### Saving Commands with Variables
+
+```
+User runs: mem save "curl -H 'Bearer eyJhbG...' https://api.example.com" -g api
+    │
+    ▼
+cli.save()
+    │ interactive terminal detected
+    ▼
+variables.detect_credentials(cmd)
+    │
+    ├── _command_may_contain_credentials() — heuristic pre-filter
+    │   checks for credential keywords + long tokens (≥16 chars)
+    │   skips simple commands (echo, ls, cd) entirely
+    │
+    ├── If pre-filter passes and apple-fm-sdk available:
+    │   │ _detect_credentials_async() via Apple FM
+    │   │   └── guided generation with CredentialList generable
+    │   │
+    │   └── _deduplicate_detections() — post-filter
+    │       removes: hallucinations, URLs, hostnames, short values, subsets
+    │       extracts: actual secret from --flag=value syntax
+    │       normalizes: CamelCase → UPPER_SNAKE_CASE
+    │
+    └── If SDK unavailable: returns empty list (save proceeds normally)
+    │
+    ▼
+User confirms/renames each detected credential
+    │ command text updated with $VAR_NAME tokens
+    ▼
+variables.parse_variables() — detect $VAR_NAME tokens
+variables.merge_var_declarations() — merge with --var flags
+    ▼
+groups.save_command()
+    │ stores command structure + var declarations (never values)
     ▼
 Done
 ```
 
+### Running Commands with Variables
+
+```
+User runs: mem run api API_TOKEN=abc123
+    │
+    ▼
+Parse inline VAR=VALUE arguments
+    ▼
+Resolve all variables upfront (before any execution):
+    1. Inline arguments  ← highest priority
+    2. Shell environment (os.environ)
+    3. Persistent store (~/.mem/vars.json)
+    4. Default value (from --var at save time)
+    5. Interactive prompt ← last resort
+    ▼
+Display resolution summary
+    ✓ $API_TOKEN resolved from arguments
+    ✓ $NAMESPACE resolved from default
+    ▼
+Execute commands with substituted values
+```
+
 ## JSONL Schemas
 
 ### Command Entry (`repos/<repo>.jsonl`)
@@ -157,7 +252,7 @@ Done
   "command": "kubectl rollout restart deployment api",
   "ts": 1709600000,
   "dir": "/Users/mati/projects/services/api",
-  "repo": "services",
+  "repo": "/Users/mati/projects/services/api",
   "exit_code": 0,
   "duration_ms": 312,
   "session": "abc123def456"
@@ -173,7 +268,7 @@ Done
   "started_at": 1709599500,
   "ended_at": 1709600400,
   "dir": "/Users/mati/projects/services/api",
-  "repo": "services",
+  "repo": "/Users/mati/projects/services/api",
   "commands": [
     "kubectl logs api-7f9b --tail=100",
     "kubectl get pods -n production",
@@ -192,14 +287,49 @@ Done
       "pattern": "kubectl get <resource>",
       "example": "kubectl get pods",
       "frequency": 42
-    },
-    {
-      "pattern": "kubectl describe <resource> <name>",
-      "example": "kubectl describe pod api-7f9b",
-      "frequency": 17
     }
   ],
-  "last_updated": 1709600000
+  "last_updated": 1709600000,
+  "processed_commands": [
+    "kubectl get pods",
+    "kubectl get services",
+    "kubectl describe pod api-7f9b"
+  ]
+}
+```
+
+### Group File (`groups/repos/<repo>.json` or `groups/_global.json`)
+
+```json
+{
+  "saved": [
+    { "cmd": "echo hello", "comment": "test" }
+  ],
+  "groups": {
+    "api": {
+      "description": "API troubleshooting",
+      "commands": [
+        {
+          "cmd": "curl -H 'Authorization: Bearer $API_TOKEN' https://api.example.com/users",
+          "comment": "list users",
+          "vars": [
+            { "name": "API_TOKEN", "default": null }
+          ]
+        }
+      ]
+    }
+  }
+}
+```
+
+### Variable Store (`vars.json`)
+
+```json
+{
+  "vars": {
+    "API_TOKEN": { "value": "sk-abc123...", "last_used": 1709600000 },
+    "DB_HOST": { "value": "staging.db.internal", "last_used": 1709500000 }
+  }
 }
 ```
 
@@ -215,5 +345,5 @@ See `docs/decisions/` for detailed ADRs:
 
 - [001: JSONL Over SQLite](docs/decisions/001-jsonl-over-sqlite.md)
 - [002: Apple FM SDK for Patterns](docs/decisions/002-apple-fm-sdk-for-patterns.md)
-- [003: No Background Daemon](docs/decisions/003-no-daemon.md)
+- [003: No Daemon, Background Subprocess Instead](docs/decisions/003-no-daemon.md)
 - [004: Per-Repository JSONL Files](docs/decisions/004-per-repo-jsonl.md)

{cli_mem-0.3.6 → cli_mem-0.4.0}/PHILOSOPHY.md

@@ -76,10 +76,13 @@ passively and speaks only when asked.
 
 - Shell hooks append one line per command execution — no
   prompts, no confirmations, no output.
-- Pattern extraction and session grouping run only when the user
-  explicitly invokes `mem sync`.
-- No background daemons, no cron jobs, no scheduled processes.
-- The user is always in control of when analysis happens.
+- Pattern extraction runs automatically in a detached background
+  process every 20 captures. Completely invisible — no stdout,
+  no stderr, no wait. The user never notices it.
+- No daemons, no cron jobs, no persistent processes. The
+  background subprocess runs, finishes, and exits on its own.
+- When the user searches or lists, results reflect the latest
+  patterns without any manual step.
 
 **Why**: A tool that nags or slows the shell will be uninstalled
 within a day. Silence is a feature.

cli_mem-0.4.0/PKG-INFO

@@ -0,0 +1,410 @@
+Metadata-Version: 2.4
+Name: cli-mem
+Version: 0.4.0
+Summary: Privacy-first CLI that turns shell history into searchable memory
+Project-URL: GitHub, https://github.com/matinsaurralde/mem
+Author: Matias Insaurralde
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: System :: Shells
+Requires-Python: >=3.10
+Requires-Dist: click
+Requires-Dist: pydantic
+Requires-Dist: rich
+Provides-Extra: ai
+Requires-Dist: apple-fm-sdk; extra == 'ai'
+Provides-Extra: dev
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: pytest-asyncio; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <h1 align="center">mem</h1>
+  <p align="center">
+    <strong>Your shell, remembered.</strong>
+  </p>
+  <p align="center">
+    A privacy-first CLI that captures, searches, and organizes your terminal history<br>
+    with on-device AI. Nothing ever leaves your machine.
+  </p>
+  <p align="center">
+    <a href="#install"><img alt="macOS 26+" src="https://img.shields.io/badge/macOS-26%2B-blue?logo=apple&logoColor=white"></a>
+    <a href="#install"><img alt="Python 3.10+" src="https://img.shields.io/badge/python-3.10%2B-3776AB?logo=python&logoColor=white"></a>
+    <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-green"></a>
+    <a href="PHILOSOPHY.md"><img alt="Privacy: on-device" src="https://img.shields.io/badge/privacy-100%25%20on--device-brightgreen"></a>
+  </p>
+</p>
+
+---
+
+## What mem does
+
+mem silently captures every command you type, then lets you search, save, and replay them — scoped to the git repo you're in.
+
+```bash
+mem deploy             # search your history
+mem save "cmd" -g ops  # save a command to a group
+mem run ops            # run the group interactively
+mem vars set API_KEY   # store a secret for saved commands
+```
+
+Unlike `Ctrl+R`, mem ranks results by frequency, recency, and the repo you're currently in. Unlike cloud-based tools, everything stays on your machine as plain text files in `~/.mem/`.
+
+---
+
+## Install
+
+```bash
+# Homebrew (recommended)
+brew install matinsaurralde/tap/mem
+
+# pip
+pip install cli-mem
+
+# With AI features (pattern extraction + credential detection)
+pip install "cli-mem[ai]"
+```
+
+Then activate the shell hook for your shell:
+
+```bash
+# zsh
+echo 'eval "$(mem init zsh)"' >> ~/.zshrc
+source ~/.zshrc
+
+# bash
+echo 'eval "$(mem init bash)"' >> ~/.bashrc
+source ~/.bashrc
+
+# fish
+echo 'mem init fish | source' >> ~/.config/fish/config.fish
+source ~/.config/fish/config.fish
+```
+
+That's it. Every command you type is now silently captured with full context (directory, git repo, exit code, duration).
+
+---
+
+## Search
+
+Just type `mem` followed by any keyword. Results are ranked by your current repo.
+
+```bash
+mem kubectl              # search by keyword
+mem "docker compose"     # search by phrase
+mem deploy -n 20         # more results
+mem deploy --json        # machine-readable output
+```
+
+```
+  1  kubectl apply -f deployment.yaml    infra       2h ago
+  2  docker compose up -d                backend     1d ago
+  3  fly deploy                          api         3d ago
+```
+
+### Patterns
+
+mem automatically learns structural patterns from your history using on-device AI. No manual step needed — extraction runs in the background every 20 commands.
+
+```bash
+mem kubectl -p
+```
+
+```
+Patterns for "kubectl":
+
+  kubectl get <resource>
+  kubectl describe <resource> <name>
+  kubectl logs <pod> [--tail=<n>]
+  kubectl apply -f <file>
+```
+
+---
+
+## Groups
+
+Groups are named collections of commands — like runbooks you can execute.
+
+### Save commands to a group
+
+```bash
+mem save "kubectl get pods -n production" --group k8s --comment "list pods"
+mem save "docker compose up -d" -g deploy -c "start services"
+```
+
+Save the last command you ran:
+
+```bash
+mem save "!" -g troubleshooting
+```
+
+### List groups
+
+```bash
+mem list                 # show all groups and saved commands
+mem list k8s             # show commands in a specific group
+mem list -g              # global scope only
+mem list -r              # current repo only
+mem list --json          # JSON output
+```
+
+### Run a group
+
+```bash
+mem run k8s              # run interactively (pick one or all)
+mem run deploy -y        # run all without prompts
+```
+
+### Manage groups
+
+```bash
+mem group rename old new       # rename a group
+mem group remove k8s           # delete a group
+mem group copy k8s --global    # copy from repo to global scope
+mem group edit k8s             # open in $EDITOR
+```
+
+### Export and import
+
+```bash
+mem export k8s                       # copy JSON to clipboard
+mem export k8s --format markdown     # copy as markdown
+mem export k8s --stdout              # print instead of clipboard
+
+mem import                           # import from clipboard (auto-detect format + group name)
+mem import -g renamed                # import from clipboard with custom group name
+mem import runbook.json -g ops       # import from file (auto-detects format)
+mem import runbook.md -g ops         # markdown works too
+```
+
+---
+
+## Variables
+
+Saved commands can contain `$VAR_NAME` placeholders that get resolved at runtime. Values never get stored in group files.
+
+### Save commands with variables
+
+```bash
+# Variables are detected automatically from $VAR_NAME tokens
+mem save "ssh -i ~/.ssh/\$KEY_NAME ubuntu@\$BASTION_HOST" -g ssh
+
+# Set a default value with --var
+mem save "kubectl get pods -n \$NAMESPACE" -g k8s --var NAMESPACE=production
+
+# AI detects hardcoded credentials and suggests variables
+mem save "curl -H 'Authorization: Bearer eyJhbGci...' https://api.example.com/users" -g api
+#  Detected possible credential: Bearer token
+#  Suggested: curl -H 'Authorization: Bearer $API_TOKEN' ...
+#  Variable name [API_TOKEN]: █
+```
+
+### Resolution priority
+
+When `mem run` encounters variables, it resolves them in this order:
+
+1. **Inline arguments** — `mem run api API_TOKEN=abc123`
+2. **Shell environment** — `export API_TOKEN=abc123`
+3. **Persistent store** — `mem vars set API_TOKEN`
+4. **Default value** — from `--var NAME=default` at save time
+5. **Interactive prompt** — asks you, only as a last resort
+
+All prompts are collected upfront before any command runs. With `--yes`, unresolved variables cause an immediate error listing what's missing.
+
+### Variable store
+
+For values that persist across sessions but shouldn't be in `.zshrc`:
+
+```bash
+mem vars set API_TOKEN           # hidden input (like sudo)
+mem vars set DB_HOST staging.db  # inline for non-sensitive values
+mem vars list                    # shows names only, never values
+mem vars remove API_TOKEN
+mem vars clear
+```
+
+### Variable status in listings
+
+`mem list` shows whether each variable is ready:
+
+```
+● backend / api
+  ──────────────────────────────────────────────────────
+  1. curl -H 'Authorization: Bearer $API_TOKEN' .../users/$USER_ID
+     ✓ $API_TOKEN  resolved from environment
+     ⚠ $USER_ID    unset — pass inline: mem run api USER_ID=42
+```
+
+---
+
+## Scoping
+
+Every group and saved command lives in either **repo scope** (tied to the current git repo) or **global scope** (available everywhere).
+
+- Inside a git repo: defaults to repo scope
+- Outside a git repo: defaults to global scope
+- Use `--global` / `-g` to force global scope
+- A repo group **shadows** a global group with the same name
+
+---
+
+## Sessions
+
+mem groups your commands into work sessions (based on 5-minute idle gaps and repo changes) so you can recall exactly what you did.
+
+```bash
+mem session "api outage"       # search sessions by keyword
+mem session debug --json       # machine-readable output
+```
+
+```
+┌ [1] Session: 2026-03-07 14:30  myapp ──────────────────┐
+│   1  kubectl logs api-7f9b --tail=100                   │
+│   2  kubectl get pods -n production                     │
+│   3  kubectl rollout restart deploy api                 │
+│   4  curl -s localhost:8080/health                      │
+└─────────────────────────────────────────────────────────┘
+
+Replay a session? [number/n]: _
+```
+
+Replaying a session executes each command with per-command confirmation.
+
+---
+
+## Other commands
+
+```bash
+mem stats                        # top commands, repos, totals
+mem stats --json                 # machine-readable stats
+mem forget "API_KEY=sk-..."      # permanently delete matching commands
+mem forget "password" --yes      # skip confirmation
+mem init zsh                     # print shell hook code (also: bash, fish)
+```
+
+---
+
+## How it works
+
+```
+You type a command
+       │
+       ▼
+  Shell hook (preexec/precmd)
+       │
+       ▼
+  mem _capture  ← runs in background, <5ms
+       │
+       ├─→ Append to ~/.mem/repos/<repo>.jsonl
+       └─→ Every 20 captures: background pattern extraction
+```
+
+**Search scoring:**
+
+```
+score = (frequency × 0.4) + (recency × 0.4) + (context × 0.2)
+```
+
+- **Frequency** — how often you've run this command
+- **Recency** — exponential decay, 7-day half-life
+- **Context** — 1.0 same repo, 0.5 same directory prefix, 0.0 otherwise
+
+**AI features** use [Apple Foundation Models](https://developer.apple.com/machine-learning/api/) running entirely on your Mac's neural engine. No API keys, no cloud, no data leaves the machine. If Apple Intelligence isn't available, everything still works — you just don't get pattern extraction or credential detection.
+
+---
+
+## Storage
+
+All data lives in `~/.mem/` as human-readable plain text:
+
+```
+~/.mem/
+  repos/
+    myapp.jsonl              # commands captured in this git repo
+    _global.jsonl            # commands outside any repo
+  sessions/
+    2026-03-07.jsonl         # work sessions by date
+  patterns/
+    kubectl.json             # AI-extracted command patterns
+    docker.json
+  groups/
+    repos/
+      myapp.json             # repo-scoped groups and saved commands
+    _global.json             # global groups and saved commands
+  vars.json                  # persistent variable store (0600 permissions)
+```
+
+Inspect anything:
+
+```bash
+cat ~/.mem/repos/myapp.jsonl
+tail -f ~/.mem/repos/myapp.jsonl    # watch commands arrive in real-time
+grep "docker" ~/.mem/repos/*.jsonl  # search across repos
+```
+
+Data rotation happens automatically in the background:
+
+| Data | Retention |
+|------|-----------|
+| Commands | 90 days |
+| Sessions | 30 days |
+| Patterns | Forever |
+
+---
+
+## Privacy
+
+- Zero network requests — not even update checks
+- Zero telemetry — no analytics, no crash reports
+- Zero cloud dependencies — fully offline, always
+- On-device AI only — runs on your Mac's neural engine
+- Plain text storage — no proprietary formats, you own your data
+
+Read more in [PHILOSOPHY.md](PHILOSOPHY.md).
+
+---
+
+## Requirements
+
+| Requirement | Version |
+|-------------|---------|
+| macOS | 26.0+ |
+| Python | 3.10+ |
+| Apple Intelligence | Optional (for patterns + credential detection) |
+
+---
+
+## Uninstall
+
+```bash
+brew uninstall mem          # or: pip uninstall cli-mem
+rm -rf ~/.mem               # remove all captured data
+```
+
+Remove the shell hook line from your shell config (`~/.zshrc`, `~/.bashrc`, or `~/.config/fish/config.fish`).
+
+---
+
+## Contributing
+
+```bash
+git clone https://github.com/matinsaurralde/mem.git
+cd mem
+pip install -e ".[dev]"
+pytest
+```
+
+Read [PHILOSOPHY.md](PHILOSOPHY.md) first.
+
+## License
+
+[MIT](LICENSE)