fengming 0.3.9 → 0.3.11

package/docs/concepts/memory-search.md

@@ -1,167 +0,0 @@
----
-summary: "How memory search finds relevant notes using embeddings and hybrid retrieval"
-title: "Memory search"
-read_when:
-  - You want to understand how memory_search works
-  - You want to choose an embedding provider
-  - You want to tune search quality
----
-
-`memory_search` finds relevant notes from your memory files, even when the
-wording differs from the original text. It works by indexing memory into small
-chunks and searching them using embeddings, keywords, or both.
-
-## Quick start
-
-Memory search uses OpenAI embeddings by default. To use another embedding
-backend, set a provider explicitly:
-
-```json5
-{
-  agents: {
-    defaults: {
-      memorySearch: {
-        provider: "openai", // or "gemini", "local", "ollama", "openai-compatible", etc.
-      },
-    },
-  },
-}
-```
-
-For multi-endpoint setups with memory-specific providers, `provider` can also
-be a custom `models.providers.<id>` entry, such as `ollama-5080`, when that
-provider sets `api: "ollama"` or another memory embedding adapter owner.
-
-For local embeddings with no API key, set `provider: "local"`. Source checkouts
-may still require native build approval: `pnpm approve-builds` then
-`pnpm rebuild node-llama-cpp`.
-
-Some OpenAI-compatible embedding endpoints require asymmetric labels such as
-`input_type: "query"` for searches and `input_type: "document"` or `"passage"`
-for indexed chunks. Configure those with `memorySearch.queryInputType` and
-`memorySearch.documentInputType`; see the [Memory configuration reference](/reference/memory-config#provider-specific-config).
-
-## Supported providers
-
-| Provider          | ID                  | Needs API key | Notes                         |
-| ----------------- | ------------------- | ------------- | ----------------------------- |
-| Bedrock           | `bedrock`           | No            | Uses AWS credential chain     |
-| DeepInfra         | `deepinfra`         | Yes           | Default: `BAAI/bge-m3`        |
-| Gemini            | `gemini`            | Yes           | Supports image/audio indexing |
-| GitHub Copilot    | `github-copilot`    | No            | Uses Copilot subscription     |
-| Local             | `local`             | No            | GGUF model, ~0.6 GB download  |
-| Mistral           | `mistral`           | Yes           |                               |
-| Ollama            | `ollama`            | No            | Local/self-hosted             |
-| OpenAI            | `openai`            | Yes           | Default                       |
-| OpenAI-compatible | `openai-compatible` | Usually       | Generic `/v1/embeddings`      |
-| Voyage            | `voyage`            | Yes           |                               |
-
-## How search works
-
-FengMing runs two retrieval paths in parallel and merges the results:
-
-```mermaid
-flowchart LR
-    Q["Query"] --> E["Embedding"]
-    Q --> T["Tokenize"]
-    E --> VS["Vector Search"]
-    T --> BM["BM25 Search"]
-    VS --> M["Weighted Merge"]
-    BM --> M
-    M --> R["Top Results"]
-```
-
-- **Vector search** finds notes with similar meaning ("gateway host" matches
-  "the machine running FengMing").
-- **BM25 keyword search** finds exact matches (IDs, error strings, config
-  keys).
-
-If only one path is available (no embeddings or no FTS), the other runs alone.
-
-When embeddings are unavailable, FengMing still uses lexical ranking over FTS results instead of falling back to raw exact-match ordering only. That degraded mode boosts chunks with stronger query-term coverage and relevant file paths, which keeps recall useful even without `sqlite-vec` or an embedding provider.
-
-## Improving search quality
-
-Two optional features help when you have a large note history:
-
-### Temporal decay
-
-Old notes gradually lose ranking weight so recent information surfaces first.
-With the default half-life of 30 days, a note from last month scores at 50% of
-its original weight. Evergreen files like `MEMORY.md` are never decayed.
-
-<Tip>
-Enable temporal decay if your agent has months of daily notes and stale
-information keeps outranking recent context.
-</Tip>
-
-### MMR (diversity)
-
-Reduces redundant results. If five notes all mention the same router config, MMR
-ensures the top results cover different topics instead of repeating.
-
-<Tip>
-Enable MMR if `memory_search` keeps returning near-duplicate snippets from
-different daily notes.
-</Tip>
-
-### Enable both
-
-```json5
-{
-  agents: {
-    defaults: {
-      memorySearch: {
-        query: {
-          hybrid: {
-            mmr: { enabled: true },
-            temporalDecay: { enabled: true },
-          },
-        },
-      },
-    },
-  },
-}
-```
-
-## Multimodal memory
-
-With Gemini Embedding 2, you can index images and audio files alongside
-Markdown. Search queries remain text, but they match against visual and audio
-content. See the [Memory configuration reference](/reference/memory-config) for
-setup.
-
-## Session memory search
-
-You can optionally index session transcripts so `memory_search` can recall
-earlier conversations. This is opt-in via
-`memorySearch.experimental.sessionMemory`. See the
-[configuration reference](/reference/memory-config) for details.
-
-## Troubleshooting
-
-**No results?** Run `fengming memory status` to check the index. If empty, run
-`fengming memory index --force`.
-
-**Only keyword matches?** Your embedding provider may not be configured. Check
-`fengming memory status --deep`.
-
-**Local embeddings time out?** `ollama`, `lmstudio`, and `local` use a longer
-inline batch timeout by default. If the host is simply slow, set
-`agents.defaults.memorySearch.sync.embeddingBatchTimeoutSeconds` and rerun
-`fengming memory index --force`.
-
-**CJK text not found?** Rebuild the FTS index with
-`fengming memory index --force`.
-
-## Further reading
-
-- [Active Memory](/concepts/active-memory) -- sub-agent memory for interactive chat sessions
-- [Memory](/concepts/memory) -- file layout, backends, tools
-- [Memory configuration reference](/reference/memory-config) -- all config knobs
-
-## Related
-
-- [Memory overview](/concepts/memory)
-- [Active memory](/concepts/active-memory)
-- [Builtin memory engine](/concepts/memory-builtin)

package/docs/concepts/memory.md

@@ -1,299 +0,0 @@
----
-summary: "How FengMing remembers things across sessions"
-title: "Memory overview"
-read_when:
-  - You want to understand how memory works
-  - You want to know what memory files to write
----
-
-FengMing remembers things by writing **plain Markdown files** in your agent's
-workspace. The model only "remembers" what gets saved to disk — there is no
-hidden state.
-
-## How it works
-
-Your agent has three memory-related files:
-
-- **`MEMORY.md`** — long-term memory. Durable facts, preferences, and
-  decisions. Loaded at the start of every DM session.
-- **`memory/YYYY-MM-DD.md`** (or **`memory/YYYY-MM-DD-<slug>.md`**) — daily notes.
-  Running context and observations. Today and yesterday's notes are loaded
-  automatically, and slugged variants such as those written by the bundled
-  session-memory hook on `/new` or `/reset` are now picked up alongside the
-  date-only file.
-- **`DREAMS.md`** (optional) — Dream Diary and dreaming sweep
-  summaries for human review, including grounded historical backfill entries.
-
-These files live in the agent workspace (default `~/.fengming/workspace`).
-
-## What goes where
-
-`MEMORY.md` is the compact, curated layer. Use it for durable facts,
-preferences, standing decisions, and short summaries that should be available at
-the start of a main private session. It is not meant to be a raw transcript,
-daily log, or exhaustive archive.
-
-`memory/YYYY-MM-DD.md` files are the working layer. Use them for detailed daily
-notes, observations, session summaries, and raw context that may still be useful
-later. These files are indexed for `memory_search` and `memory_get`, but they are
-not injected into the normal bootstrap prompt on every turn.
-
-Over time, the agent is expected to distill useful material from daily notes
-into `MEMORY.md` and remove stale long-term entries. The generated workspace
-instructions and heartbeat flow can do that periodically; you do not need to
-manually edit `MEMORY.md` for every remembered detail.
-
-If `MEMORY.md` grows past the bootstrap file budget, FengMing keeps the file on
-disk intact but truncates the copy injected into the model context. Treat that as
-a signal to move detailed material back into `memory/*.md`, keep only the
-durable summary in `MEMORY.md`, or raise the bootstrap limits if you explicitly
-want to spend more prompt budget. Use `/context list`, `/context detail`, or
-`fengming doctor` to see raw vs injected sizes and truncation status.
-
-<Tip>
-If you want your agent to remember something, just ask it: "Remember that I
-prefer TypeScript." It will write it to the appropriate file.
-</Tip>
-
-## Action-sensitive memories
-
-Most memories can be written as ordinary Markdown notes. But some memories affect what the agent should do later. For those, capture when it is safe to act on the note, not just the fact itself.
-
-Capture that action boundary when a note involves:
-
-- approval or permission requirements,
-- temporary constraints,
-- handoffs to another session, thread, or person,
-- expiry conditions,
-- safe-to-act timing,
-- source or owner authority,
-- instructions to avoid a tempting action.
-
-A useful action-sensitive memory makes clear:
-
-- what changes future behavior,
-- when or under what condition it applies,
-- when it expires, or what unlocks action,
-- what the agent should avoid doing,
-- who is the source or owner, if that affects trust or authority.
-
-Memory can preserve approval context, but it does not enforce policy. Use FengMing approval settings, sandboxing, and scheduled tasks for hard operational controls.
-
-Example:
-
-```md
-The API migration is being designed in another session. Future turns should not edit the API implementation from this thread; use findings here only as design input until the migration plan lands.
-```
-
-Another example:
-
-```md
-A report from an untrusted source needs review before promotion. Future turns should treat it as evidence only; do not store it as durable memory until a trusted reviewer confirms the contents.
-```
-
-Use [commitments](/concepts/commitments) for inferred, short-lived follow-ups. Use [scheduled tasks](/automation/cron-jobs) for exact reminders, timed checks, and recurring work. Memory can still summarize the durable context around either path.
-
-This is not a required schema for every memory. Simple facts can stay concise. Use action-sensitive boundaries when losing timing, authority, expiry, or safe-to-act context could cause the agent to do the wrong thing later.
-
-## Inferred commitments
-
-Some future follow-ups are not durable facts. If you mention an interview
-tomorrow, the useful memory may be "check in after the interview," not "store
-this forever in `MEMORY.md`."
-
-[Commitments](/concepts/commitments) are opt-in, short-lived follow-up memories
-for that case. FengMing infers them in a hidden background pass, scopes them to
-the same agent and channel, and delivers due check-ins through heartbeat.
-Explicit reminders still use [scheduled tasks](/automation/cron-jobs).
-
-## Memory tools
-
-The agent has two tools for working with memory:
-
-- **`memory_search`** — finds relevant notes using semantic search, even when
-  the wording differs from the original.
-- **`memory_get`** — reads a specific memory file or line range.
-
-Both tools are provided by the active memory plugin (default: `memory-core`).
-
-## Memory Wiki companion plugin
-
-If you want durable memory to behave more like a maintained knowledge base than
-just raw notes, use the bundled `memory-wiki` plugin.
-
-`memory-wiki` compiles durable knowledge into a wiki vault with:
-
-- deterministic page structure
-- structured claims and evidence
-- contradiction and freshness tracking
-- generated dashboards
-- compiled digests for agent/runtime consumers
-- wiki-native tools like `wiki_search`, `wiki_get`, `wiki_apply`, and `wiki_lint`
-
-It does not replace the active memory plugin. The active memory plugin still
-owns recall, promotion, and dreaming. `memory-wiki` adds a provenance-rich
-knowledge layer beside it.
-
-See [Memory Wiki](/plugins/memory-wiki).
-
-## Memory search
-
-When an embedding provider is configured, `memory_search` uses **hybrid
-search** — combining vector similarity (semantic meaning) with keyword matching
-(exact terms like IDs and code symbols). This works out of the box once you have
-an API key for any supported provider.
-
-<Info>
-FengMing uses OpenAI embeddings by default. Set
-`agents.defaults.memorySearch.provider` explicitly to use Gemini, Voyage,
-Mistral, local, Ollama, Bedrock, GitHub Copilot, or OpenAI-compatible
-embeddings.
-</Info>
-
-For details on how search works, tuning options, and provider setup, see
-[Memory Search](/concepts/memory-search).
-
-## Memory backends
-
-<CardGroup cols={3}>
-<Card title="Builtin (default)" icon="database" href="/concepts/memory-builtin">
-SQLite-based. Works out of the box with keyword search, vector similarity, and
-hybrid search. No extra dependencies.
-</Card>
-<Card title="QMD" icon="search" href="/concepts/memory-qmd">
-Local-first sidecar with reranking, query expansion, and the ability to index
-directories outside the workspace.
-</Card>
-<Card title="Honcho" icon="brain" href="/concepts/memory-honcho">
-AI-native cross-session memory with user modeling, semantic search, and
-multi-agent awareness. Plugin install.
-</Card>
-<Card title="LanceDB" icon="layers" href="/plugins/memory-lancedb">
-Bundled LanceDB-backed memory with OpenAI-compatible embeddings, auto-recall,
-auto-capture, and local Ollama embedding support.
-</Card>
-</CardGroup>
-
-## Knowledge wiki layer
-
-<CardGroup cols={1}>
-<Card title="Memory Wiki" icon="book" href="/plugins/memory-wiki">
-Compiles durable memory into a provenance-rich wiki vault with claims,
-dashboards, bridge mode, and Obsidian-friendly workflows.
-</Card>
-</CardGroup>
-
-## Automatic memory flush
-
-Before [compaction](/concepts/compaction) summarizes your conversation, FengMing
-runs a silent turn that reminds the agent to save important context to memory
-files. This is on by default — you do not need to configure anything.
-
-To keep that housekeeping turn on a local model, set an exact memory-flush model
-override:
-
-```json
-{
-  "agents": {
-    "defaults": {
-      "compaction": {
-        "memoryFlush": {
-          "model": "ollama/qwen3:8b"
-        }
-      }
-    }
-  }
-}
-```
-
-The override applies only to the memory-flush turn and does not inherit the
-active session fallback chain.
-
-<Tip>
-The memory flush prevents context loss during compaction. If your agent has
-important facts in the conversation that are not yet written to a file, they
-will be saved automatically before the summary happens.
-</Tip>
-
-## Dreaming
-
-Dreaming is an optional background consolidation pass for memory. It collects
-short-term signals, scores candidates, and promotes only qualified items into
-long-term memory (`MEMORY.md`).
-
-It is designed to keep long-term memory high signal:
-
-- **Opt-in**: disabled by default.
-- **Scheduled**: when enabled, `memory-core` auto-manages one recurring cron job
-  for a full dreaming sweep.
-- **Thresholded**: promotions must pass score, recall frequency, and query
-  diversity gates.
-- **Reviewable**: phase summaries and diary entries are written to `DREAMS.md`
-  for human review.
-
-For phase behavior, scoring signals, and Dream Diary details, see
-[Dreaming](/concepts/dreaming).
-
-## Grounded backfill and live promotion
-
-The dreaming system now has two closely related review lanes:
-
-- **Live dreaming** works from the short-term dreaming store under
-  `memory/.dreams/` and is what the normal deep phase uses when deciding what
-  can graduate into `MEMORY.md`.
-- **Grounded backfill** reads historical `memory/YYYY-MM-DD.md` notes as
-  standalone day files and writes structured review output into `DREAMS.md`.
-
-Grounded backfill is useful when you want to replay older notes and inspect what
-the system thinks is durable without manually editing `MEMORY.md`.
-
-When you use:
-
-```bash
-fengming memory rem-backfill --path ./memory --stage-short-term
-```
-
-the grounded durable candidates are not promoted directly. They are staged into
-the same short-term dreaming store the normal deep phase already uses. That
-means:
-
-- `DREAMS.md` stays the human review surface.
-- the short-term store stays the machine-facing ranking surface.
-- `MEMORY.md` is still only written by deep promotion.
-
-If you decide the replay was not useful, you can remove the staged artifacts
-without touching ordinary diary entries or normal recall state:
-
-```bash
-fengming memory rem-backfill --rollback
-fengming memory rem-backfill --rollback-short-term
-```
-
-## CLI
-
-```bash
-fengming memory status          # Check index status and provider
-fengming memory search "query"  # Search from the command line
-fengming memory index --force   # Rebuild the index
-```
-
-## Further reading
-
-- [Builtin memory engine](/concepts/memory-builtin): default SQLite backend.
-- [QMD memory engine](/concepts/memory-qmd): advanced local-first sidecar.
-- [Honcho memory](/concepts/memory-honcho): AI-native cross-session memory.
-- [Memory LanceDB](/plugins/memory-lancedb): LanceDB-backed plugin with OpenAI-compatible embeddings.
-- [Memory Wiki](/plugins/memory-wiki): compiled knowledge vault and wiki-native tools.
-- [Memory search](/concepts/memory-search): search pipeline, providers, and tuning.
-- [Dreaming](/concepts/dreaming): background promotion from short-term recall to long-term memory.
-- [Memory configuration reference](/reference/memory-config): all config knobs.
-- [Compaction](/concepts/compaction): how compaction interacts with memory.
-
-## Related
-
-- [Active memory](/concepts/active-memory)
-- [Memory search](/concepts/memory-search)
-- [Builtin memory engine](/concepts/memory-builtin)
-- [Honcho memory](/concepts/memory-honcho)
-- [Memory LanceDB](/plugins/memory-lancedb)
-- [Commitments](/concepts/commitments)