@rekal/mem 0.0.0

package/test/fixtures/store/concepts/memory-coherence.md

@@ -0,0 +1,33 @@
+---
+description: "Memory coherence — ensuring implanted memories integrate without contradicting existing ones"
+tags: [neuroscience, memory, coherence]
+entities: [Rekall Inc]
+---
+
+# Memory Coherence
+
+Memory coherence is the measure of how well an implanted memory integrates with a subject's existing memory topology. High coherence means the implant feels natural and indistinguishable from genuine memories. Low coherence leads to rejection cascades.
+
+## Coherence Score
+
+Measured on a 0-100 scale:
+- **90-100:** Seamless integration, indistinguishable from genuine memory
+- **70-89:** Minor artifacts, occasional deja vu sensations
+- **40-69:** Noticeable inconsistencies, risk of lucid bleed-through
+- **Below 40:** Emergency extraction recommended, ego fragmentation risk
+
+## Rejection Cascades
+
+When an implanted memory contradicts a genuine one, the brain attempts to resolve the conflict. If unmanaged, this creates a cascade:
+
+1. **Micro-contradiction:** Small detail mismatch (e.g., wrong hand dominance in implant)
+2. **Temporal paradox:** Implanted timeline conflicts with genuine timeline
+3. **Identity conflict:** Implanted persona contradicts core identity beliefs
+4. **Cascade:** Multiple contradictions reinforce each other, destabilizing the memory topology
+
+## Prevention
+
+- Pre-implant neural mapping identifies high-risk memory regions
+- Contextual anchoring ties implants to genuine emotional experiences
+- Gradual integration over multiple sessions for complex implants
+- Real-time monitoring with automatic dampening during the procedure

package/test/fixtures/store/concepts/rag.md

@@ -0,0 +1,27 @@
+---
+description: "Retrieval-Augmented Generation — fetch relevant context before generating a response"
+tags: [ai, retrieval, rag, architecture]
+entities: [RAG, Rekall Inc]
+---
+
+# Retrieval-Augmented Generation (RAG)
+
+RAG augments LLM generation by retrieving relevant documents from an external store and injecting them into the prompt. Reduces hallucination and enables knowledge that wasn't in training data.
+
+## Pipeline
+
+1. **Query:** user message or derived search query
+2. **Retrieve:** search a document store (vector similarity, BM25, or hybrid)
+3. **Augment:** inject retrieved documents into the LLM prompt as context
+4. **Generate:** LLM produces a response grounded in the retrieved context
+
+## Limitations
+
+- Retrieval quality bottlenecks generation quality
+- Chunk boundaries can split relevant context
+- Token budget limits how much context can be injected
+- Flat retrieval misses hierarchical relationships between documents
+
+## Evolution
+
+Rekall's early retrieval system used basic vector-only search. The current platform improves on this with hybrid BM25+vector retrieval, hierarchical document structure, and frecency-based injection that prioritizes contextually relevant memories over raw similarity scores.

package/test/fixtures/store/index.md

@@ -0,0 +1,9 @@
+---
+description: "Root of Douglas Quaid's memory store — projects, concepts, and agent configuration"
+tags: [root]
+entities: [Douglas Quaid]
+---
+
+# Memory Store
+
+Personal knowledge base for Douglas Quaid, covering Rekall Inc projects, technical concepts, and agent configuration.

package/test/fixtures/store/projects/index.md

@@ -0,0 +1,9 @@
+---
+description: "Douglas's projects — Rekall Inc (memory implant platform), Mars Colony OS"
+tags: [projects]
+entities: [Douglas Quaid, Rekall Inc, Mars Colony OS]
+---
+
+# Projects
+
+Active and past projects spanning memory technology and colonial infrastructure. Current focus is Rekall Inc's memory implant platform.

package/test/fixtures/store/projects/rekall-inc/architecture.md

@@ -0,0 +1,41 @@
+---
+description: "Rekall memory implant architecture — engram pipeline, coherence engine, and neural interface"
+tags: [architecture, memory, neuroscience]
+entities: [Rekall Inc, Engram Pipeline]
+---
+
+# Rekall Architecture
+
+The memory implant system consists of three layers: the neural interface (hardware), the engram pipeline (synthesis), and the coherence engine (validation).
+
+## Neural Interface
+
+The chair-mounted neural interface uses quantum resonance imaging to map the client's existing memory topology. This map serves as the scaffold for implant integration.
+
+- Non-invasive transcranial stimulation
+- 12-minute mapping session
+- Compatible with 99.7% of neural architectures
+
+## Engram Pipeline
+
+Raw experience data is synthesized into engrams — self-consistent memory units that integrate with the client's existing neural patterns.
+
+1. **Capture:** Record or generate base experience data
+2. **Personalize:** Adapt sensory details to client's neural profile
+3. **Contextualize:** Weave in emotional responses and temporal anchors
+4. **Compress:** Optimize for cortical buffer storage limits
+
+## Coherence Engine
+
+The coherence engine monitors implanted memories for integration stability. Rejection cascades occur when implanted memories conflict with genuine ones.
+
+- Real-time conflict detection during implantation
+- Post-implant monitoring for 72 hours
+- Automatic dampening of contradiction signals
+- Emergency extraction protocol if coherence drops below 40%
+
+## Known Issues
+
+- Ego death risk at 0.003% for identity overlay services
+- Lucid bleed-through in 12% of vacation implants (client becomes aware the memory is synthetic)
+- Mars packages have elevated rejection rates due to atmospheric sensory mismatch

package/test/fixtures/store/projects/rekall-inc/decisions/index.md

@@ -0,0 +1,9 @@
+---
+description: "Architecture decisions for Rekall Inc — ethics, technology choices, safety protocols"
+tags: [decisions, architecture]
+entities: [Rekall Inc]
+---
+
+# Rekall Architecture Decisions
+
+Records of significant design and policy decisions, documenting the reasoning behind key choices in memory implant technology and business operations.

package/test/fixtures/store/projects/rekall-inc/decisions/no-military.md

@@ -0,0 +1,20 @@
+---
+description: "Decision: Rekall does not provide military memory implants — ethical and liability concerns"
+tags: [decision, ethics, policy]
+entities: [Rekall Inc, Cohaagen]
+---
+
+# Decision: No Military Memory Implants
+
+Rekall does not offer combat training or tactical memory implants to military clients, despite significant revenue potential.
+
+## Rationale
+
+- Combat memories carry high rejection cascade risk due to extreme emotional content
+- Liability exposure for implanted skills used in real combat situations
+- Ethical concerns about creating soldiers with artificial experience
+- Regulatory scrutiny from the Colonial Authority
+
+## Consequence
+
+Cohaagen has pushed back on this repeatedly, citing government contract opportunities. The board maintains the policy but reviews it annually. Skill implants for self-defense (civilian grade) remain available.

package/test/fixtures/store/projects/rekall-inc/index.md

@@ -0,0 +1,28 @@
+---
+description: "Rekall Inc — memory implant platform for vacation experiences, skill acquisition, and identity services"
+tags: [project, ai, memory, neuroscience]
+entities: [Rekall Inc, Douglas Quaid, Dr. Edgemar, Cohaagen]
+---
+
+# Rekall Inc
+
+Memory implant company specializing in artificial experience creation. Clients choose from vacation packages, skill implants, and custom identity overlays. Founded in 2078, headquartered in Chicago.
+
+## Services
+
+- **Vacation memories:** Experience Mars, Saturn's rings, or deep ocean exploration without leaving your chair
+- **Skill acquisition:** Instant expertise in languages, martial arts, or technical fields
+- **Identity services:** Temporary persona overlays for entertainment or therapeutic purposes
+
+## Technology Stack
+
+- Neural mapping via quantum resonance imaging
+- Engram synthesis using transformer-based memory models
+- Real-time coherence validation to prevent rejection cascades
+- Distributed storage across redundant cortical buffers
+
+## Team
+
+- **Dr. Edgemar** — Chief Science Officer, pioneer in synthetic engram research
+- **Douglas Quaid** — Lead Engineer, memory coherence and retrieval systems
+- **Cohaagen** — CEO, business strategy and government contracts

package/test/fixtures/store/user/family.md

@@ -0,0 +1,13 @@
+---
+description: "Douglas's family — wife Lori, recurring dreams about a woman named Melina on Mars"
+tags: [user, family, personal]
+entities: [Douglas Quaid, Lori Quaid, Melina]
+---
+
+# Douglas's Family
+
+Douglas lives with his wife Lori in their apartment in Chicago. Married for several years, though Douglas has been having recurring dreams about Mars and a mysterious woman named Melina.
+
+- **Wife:** Lori Quaid
+- **Home:** Chicago, apartment near the transit hub
+- **Recurring dream:** A woman named Melina, red Martian sky, a feeling of being someone else entirely

package/test/fixtures/store/user/index.md

@@ -0,0 +1,9 @@
+---
+description: "Personal information about Douglas Quaid — family, profile, preferences"
+tags: [user, personal]
+entities: [Douglas Quaid]
+---
+
+# Douglas Quaid — Personal Information
+
+Information about Douglas Quaid: family, career at Rekall Inc, coding preferences, and background.

package/test/fixtures/store/user/preferences.md

@@ -0,0 +1,29 @@
+---
+description: "Douglas's working style — hands-on debugging, visual thinker, prefers simple solutions"
+tags: [user, preferences, workflow]
+entities: [Douglas Quaid]
+---
+
+# Working Style and Preferences
+
+Douglas is a hands-on engineer who prefers debugging by inspection over abstract analysis. Visual thinker, whiteboard-first approach.
+
+## Development Philosophy
+
+- Build it, run it, fix it — iterate fast
+- Prefer simple solutions over clever ones
+- If you can't explain it on a whiteboard, it's too complex
+- Test with real neural data, not synthetic benchmarks
+
+## Coding Style
+
+- 2-space indentation
+- TypeScript for platform code, Rust for neural interface drivers
+- Comments for "why," not "what"
+- Line width 100
+
+## Tools
+
+- Neovim as primary editor
+- Fish shell
+- Debugs memory coherence issues by visualizing engram topology graphs

package/test/fixtures/store/user/profile.md

@@ -0,0 +1,29 @@
+---
+description: "Douglas's identity — construction worker turned engineer, obsessed with Mars, memory anomalies"
+tags: [user, identity, career]
+entities: [Douglas Quaid, Rekall Inc, Mars Colony]
+---
+
+# Douglas Quaid
+
+Douglas Quaid is a construction worker in Chicago who recently transitioned to engineering at Rekall Inc. Plagued by vivid dreams about Mars that feel more like memories than imagination.
+
+## Background
+
+- Construction worker for 8 years, specializing in heavy machinery
+- Self-taught programmer, picked up neural interface engineering
+- Recruited by Rekall Inc for his intuitive understanding of memory coherence
+- Based in Chicago
+
+## Career at Rekall
+
+- Lead Engineer on the memory coherence and retrieval systems
+- Designed the conflict detection algorithm for the coherence engine
+- Skeptical of the identity overlay service — thinks it's "playing with fire"
+
+## The Mars Question
+
+- Recurring dreams about Mars since childhood
+- Dreams feature specific locations: Venusville, the reactor, the mines
+- Visited Rekall as a client before becoming an employee — session was interrupted
+- Can't shake the feeling that the dreams are real memories, not imagination

package/test/fs.test.ts

@@ -0,0 +1,15 @@
+import { describe, expect, test } from "vitest"
+import { normPath } from "../src/fs.ts"
+
+describe("norm", () => {
+  test("expands tilde to homedir", () => {
+    const result = normPath("~/test")
+    expect(result).not.toContain("~")
+    expect(result).toMatch(/^\//)
+  })
+
+  test("resolves relative paths", () => {
+    const result = normPath("./relative/path")
+    expect(result).toMatch(/^\//)
+  })
+})

package/test/glob.test.ts

@@ -0,0 +1,190 @@
+import { fileURLToPath } from "node:url"
+import { join } from "pathe"
+import { describe, expect, test } from "vitest"
+import { glob } from "../src/glob.ts"
+
+const FIXTURES = join(fileURLToPath(import.meta.url), "..", "fixtures/store")
+
+async function collect(opts: Partial<Parameters<typeof glob>[0]> = {}) {
+  const results: string[] = []
+  for await (const path of glob({ cwd: FIXTURES, ...opts })) {
+    results.push(path)
+  }
+  return results
+}
+
+describe("glob", () => {
+  test("yields files and directories", async () => {
+    const results = await collect()
+    expect(results.length).toBeGreaterThan(0)
+    expect(results.some((r) => r.endsWith(".md"))).toBe(true)
+    expect(results.some((r) => r.endsWith("/"))).toBe(true)
+  })
+
+  test("finds known fixture files", async () => {
+    const results = await collect()
+    expect(results).toContain("index.md")
+    expect(results).toContain("user/")
+    expect(results).toContain("user/family.md")
+    expect(results).toContain("user/index.md")
+    expect(results).toContain("projects/rekall-inc/decisions/no-military.md")
+  })
+
+  test("results are sorted by name", async () => {
+    const results = await collect()
+    const topLevelFiles = results.filter((r) => !r.includes("/") && r.endsWith(".md"))
+    const sorted = [...topLevelFiles].toSorted()
+    expect(topLevelFiles).toEqual(sorted)
+  })
+
+  test("respects depth", async () => {
+    const results = await collect({ depth: 2 })
+    expect(results).toContain("index.md")
+    expect(results).toContain("user/")
+    expect(results.some((r) => r.startsWith("user/") && r.endsWith(".md"))).toBe(true)
+    expect(results.some((r) => r.includes("decisions/"))).toBe(false)
+  })
+
+  test("depth 1 only shows root contents", async () => {
+    const results = await collect({ depth: 1 })
+    expect(results).toContain("index.md")
+    expect(results).toContain("user/")
+    expect(results.every((r) => !r.includes("/") || r.endsWith("/"))).toBe(true)
+  })
+
+  test("skips hidden files and dirs by default", async () => {
+    const results = await collect()
+    expect(results.every((r) => !r.startsWith("."))).toBe(true)
+    expect(results.every((r) => !r.includes("/."))).toBe(true)
+    expect(results.some((r) => r.includes(".rekal"))).toBe(false)
+  })
+
+  test("includes hidden files when hidden is true", async () => {
+    const results = await collect({ depth: 1, hidden: true })
+    expect(results.length).toBeGreaterThan(0)
+  })
+
+  test("type=files excludes directories from output", async () => {
+    const results = await collect({ type: "file" })
+    expect(results.every((r) => !r.endsWith("/"))).toBe(true)
+    expect(results.some((r) => r.includes("/"))).toBe(true)
+  })
+})
+
+describe("glob patterns", () => {
+  test("filters to matching files only", async () => {
+    const results = await collect({ glob: "**/*.md" })
+    const files = results.filter((r) => !r.endsWith("/"))
+    expect(files.length).toBeGreaterThan(0)
+    expect(files.every((r) => r.endsWith(".md"))).toBe(true)
+  })
+
+  test("excludes non-matching files", async () => {
+    const results = await collect({ glob: "**/*.md" })
+    expect(results.some((r) => r.endsWith(".yaml"))).toBe(false)
+  })
+
+  test("still includes directories for traversal", async () => {
+    const results = await collect({ glob: "**/*.md" })
+    expect(results.some((r) => r.endsWith("/"))).toBe(true)
+  })
+
+  test("supports multiple glob patterns", async () => {
+    const results = await collect({ glob: ["**/family.md", "**/profile.md"] })
+    expect(results.some((r) => r.endsWith("family.md"))).toBe(true)
+    expect(results.some((r) => r.endsWith("profile.md"))).toBe(true)
+  })
+
+  test("narrow glob reduces results", async () => {
+    const allMd = await collect({ glob: "**/*.md" })
+    const userOnly = await collect({ glob: "user/*.md" })
+    expect(userOnly.length).toBeLessThan(allMd.length)
+    const files = userOnly.filter((r) => !r.endsWith("/"))
+    expect(files.every((r) => r.startsWith("user/"))).toBe(true)
+  })
+})
+
+describe("glob empty directories", () => {
+  test("excludes empty directories by default", async () => {
+    const results = await collect()
+    expect(results.some((r) => r === "empty-dir/")).toBe(false)
+  })
+
+  test("includes empty directories when empty is true", async () => {
+    const results = await collect({ empty: true })
+    expect(results.some((r) => r === "empty-dir/")).toBe(true)
+  })
+})
+
+describe("glob onVisit", () => {
+  test("calls onVisit for entries", async () => {
+    const visited: string[] = []
+    await collect({ onVisit: (rel) => visited.push(rel) })
+    expect(visited.length).toBeGreaterThan(0)
+  })
+})
+
+describe("glob onError", () => {
+  test("calls onError for inaccessible directories", async () => {
+    const errors: { path: string; error: Error }[] = []
+    await collect({
+      cwd: "/nonexistent-path-that-does-not-exist",
+      onError: (path, error) => errors.push({ error, path }),
+    })
+    expect(errors.length).toBeGreaterThan(0)
+  })
+})
+
+describe("glob sort", () => {
+  test("type sort puts directories before files", async () => {
+    const results = await collect({ depth: 1, sort: "type" })
+    const firstFile = results.findIndex((r) => !r.endsWith("/"))
+    const lastDir = results.findLastIndex((r) => r.endsWith("/"))
+    if (firstFile !== -1 && lastDir !== -1) {
+      expect(lastDir).toBeLessThan(firstFile)
+    }
+  })
+
+  test("name sort is alphabetical", async () => {
+    const results = await collect({ depth: 1, sort: "name" })
+    const sorted = [...results].toSorted()
+    expect(results).toEqual(sorted)
+  })
+})
+
+describe("glob ignore", () => {
+  test("respects exclude rules", async () => {
+    const results = await collect({ exclude: ["user/"] })
+    expect(results.some((r) => r.startsWith("user/"))).toBe(false)
+    expect(results.some((r) => r === "user/")).toBe(false)
+    expect(results.some((r) => r.startsWith("projects/"))).toBe(true)
+  })
+
+  test("ignore=false skips all ignore file processing", async () => {
+    const withIgnore = await collect()
+    const withoutIgnore = await collect({ hidden: true, ignore: false })
+    expect(withoutIgnore.length).toBeGreaterThanOrEqual(withIgnore.length)
+  })
+})
+
+describe("glob nested ignore files", () => {
+  const IGNORE_FIXTURES = join(fileURLToPath(import.meta.url), "..", "fixtures/ignore-test")
+
+  async function collectIgnore(opts: Partial<Parameters<typeof glob>[0]> = {}) {
+    const results: string[] = []
+    for await (const path of glob({ cwd: IGNORE_FIXTURES, ...opts })) {
+      results.push(path)
+    }
+    return results
+  }
+
+  test("nested gitignore with / prefix only applies relative to its directory", async () => {
+    // sub/.gitignore contains "/skip.log"
+    // This should ignore sub/skip.log but NOT skip.log at the root
+    const results = await collectIgnore({ hidden: false })
+    expect(results).toContain("skip.log") // root skip.log should NOT be ignored
+    expect(results).not.toContain("sub/skip.log") // sub/skip.log SHOULD be ignored
+    expect(results).toContain("keep.md")
+    expect(results).toContain("sub/keep.md")
+  })
+})

package/test/md.test.ts

@@ -0,0 +1,177 @@
+import { describe, expect, test } from "vitest"
+import { chunkMarkdown, parseSections } from "../src/md.ts"
+
+describe("parse", () => {
+  test("parses sections by heading", () => {
+    const sections = parseSections(`# Title
+
+Intro paragraph.
+
+## Section One
+
+Content one.
+
+## Section Two
+
+Content two.
+`)
+    expect(sections).toHaveLength(3)
+    expect(sections[0].headingText).toBe("# Title")
+    expect(sections[0].level).toBe(1)
+    expect(sections[1].headingText).toBe("## Section One")
+    expect(sections[1].level).toBe(2)
+    expect(sections[2].headingText).toBe("## Section Two")
+    expect(sections[2].level).toBe(2)
+  })
+
+  test("handles content before first heading", () => {
+    const sections = parseSections(`Some preamble text.
+
+# First Heading
+
+Content.
+`)
+    expect(sections).toHaveLength(2)
+    expect(sections[0].headingText).toBe("")
+    expect(sections[0].level).toBe(0)
+    expect(sections[0].content.join("\n")).toContain("preamble")
+  })
+
+  test("ignores headings inside code blocks", () => {
+    const sections = parseSections(`# Real Heading
+
+\`\`\`markdown
+# Not a heading
+## Also not a heading
+\`\`\`
+
+More content.
+`)
+    expect(sections).toHaveLength(1)
+    expect(sections[0].headingText).toBe("# Real Heading")
+    expect(sections[0].content.join("\n")).toContain("# Not a heading")
+  })
+
+  test("handles tilde code fences", () => {
+    const sections = parseSections(`# Title
+
+~~~python
+# comment not a heading
+~~~
+
+## Next Section
+
+Content.
+`)
+    expect(sections).toHaveLength(2)
+    expect(sections[0].headingText).toBe("# Title")
+    expect(sections[1].headingText).toBe("## Next Section")
+  })
+
+  test("discards empty sections", () => {
+    // A section with a heading but no content lines (immediately followed by another heading)
+    const sections = parseSections(`# First
+## Second
+
+Content here.
+`)
+    // "First" has "## Second" as its only content line before being replaced,
+    // but parse() keeps sections that have the heading line itself as content
+    expect(sections).toHaveLength(2)
+    expect(sections[0].headingText).toBe("# First")
+    expect(sections[1].headingText).toBe("## Second")
+  })
+
+  test("handles nested heading levels", () => {
+    const sections = parseSections(`# H1
+
+## H2
+
+### H3
+
+Content.
+
+## Another H2
+
+More content.
+`)
+    expect(sections.map((s) => s.headingText)).toEqual(["# H1", "## H2", "### H3", "## Another H2"])
+    expect(sections.map((s) => s.level)).toEqual([1, 2, 3, 2])
+  })
+})
+
+describe("chunk", () => {
+  // Simple mock tokenizer that counts words as tokens
+  const mockTokenizer = {
+    toks: (text: string) => text.split(/\s+/).filter(Boolean).length,
+  } as Parameters<typeof chunkMarkdown>[1]
+
+  test("small document produces single chunk", () => {
+    const result = chunkMarkdown("# Title\n\nShort content.", mockTokenizer, 100)
+    expect(result).toHaveLength(1)
+    expect(result[0]).toContain("Title")
+    expect(result[0]).toContain("Short content.")
+  })
+
+  test("splits at section boundaries", () => {
+    const md = `# Title
+
+First section with enough words to take some space in the chunk budget.
+
+## Second Section
+
+Second section content that should go into a new chunk because of size limits.
+`
+    const result = chunkMarkdown(md, mockTokenizer, 15)
+    expect(result.length).toBeGreaterThan(1)
+  })
+
+  test("includes parent headings in child chunks", () => {
+    const md = `# Parent
+
+## Child
+
+Child content that is long enough to be its own chunk separate from the parent section.
+`
+    const result = chunkMarkdown(md, mockTokenizer, 15)
+    // The child chunk should include "# Parent" for context
+    const childChunk = result.find((c) => c.includes("Child content"))
+    expect(childChunk).toContain("# Parent")
+  })
+
+  test("handles deeply nested headings", () => {
+    const md = `# Level 1
+
+## Level 2
+
+### Level 3
+
+Deep content that needs all its parent headings for context.
+`
+    const result = chunkMarkdown(md, mockTokenizer, 20)
+    const deepChunk = result.find((c) => c.includes("Deep content"))
+    expect(deepChunk).toBeDefined()
+    if (deepChunk) {
+      expect(deepChunk).toContain("# Level 1")
+      expect(deepChunk).toContain("## Level 2")
+    }
+  })
+
+  test("does not duplicate headings when packing siblings", () => {
+    const md = `# Parent
+
+## Child A
+
+Short A.
+
+## Child B
+
+Short B.
+`
+    const result = chunkMarkdown(md, mockTokenizer, 100)
+    // Both children fit in one chunk, parent heading should appear only once
+    expect(result).toHaveLength(1)
+    const count = result[0].split("# Parent").length - 1
+    expect(count).toBe(1)
+  })
+})