basic-memory 0.2.21__tar.gz → 0.4.0__tar.gz

{basic_memory-0.2.21 → basic_memory-0.4.0}/.gitignore

@@ -42,3 +42,4 @@ ENV/
 
 # macOS
 .DS_Store
+/.coverage.*

{basic_memory-0.2.21 → basic_memory-0.4.0}/CHANGELOG.md

@@ -1,6 +1,33 @@
 # CHANGELOG
 
 
+## v0.4.0 (2025-02-16)
+
+### Features
+
+- Import chatgpt conversation data ([#9](https://github.com/basicmachines-co/basic-memory/pull/9),
+  [`56f47d6`](https://github.com/basicmachines-co/basic-memory/commit/56f47d6812982437f207629e6ac9a82e0e56514e))
+
+Co-authored-by: phernandez <phernandez@basicmachines.co>
+
+- Import claude.ai data ([#8](https://github.com/basicmachines-co/basic-memory/pull/8),
+  [`a15c346`](https://github.com/basicmachines-co/basic-memory/commit/a15c346d5ebd44344b76bad877bb4d1073fcbc3b))
+
+Import Claude.ai conversation and project data to basic-memory Markdown format.
+
+---------
+
+Co-authored-by: phernandez <phernandez@basicmachines.co>
+
+
+## v0.3.0 (2025-02-15)
+
+### Bug Fixes
+
+- Refactor db schema migrate handling
+  ([`ca632be`](https://github.com/basicmachines-co/basic-memory/commit/ca632beb6fed5881f4d8ba5ce698bb5bc681e6aa))
+
+
 ## v0.2.21 (2025-02-15)
 
 ### Bug Fixes
@@ -8,6 +35,17 @@
 - Fix osx installer github action
   ([`65ebe5d`](https://github.com/basicmachines-co/basic-memory/commit/65ebe5d19491e5ff047c459d799498ad5dd9cd1a))
 
+- Handle memory:// url format in read_note tool
+  ([`e080373`](https://github.com/basicmachines-co/basic-memory/commit/e0803734e69eeb6c6d7432eea323c7a264cb8347))
+
+- Remove create schema from init_db
+  ([`674dd1f`](https://github.com/basicmachines-co/basic-memory/commit/674dd1fd47be9e60ac17508476c62254991df288))
+
+### Features
+
+- Set version in var, output version at startup
+  ([`a91da13`](https://github.com/basicmachines-co/basic-memory/commit/a91da1396710e62587df1284da00137d156fc05e))
+
 
 ## v0.2.20 (2025-02-14)
 

{basic_memory-0.2.21 → basic_memory-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: basic-memory
-Version: 0.2.21
+Version: 0.4.0
 Summary: Local-first knowledge management combining Zettelkasten with knowledge graphs
 Project-URL: Homepage, https://github.com/basicmachines-co/basic-memory
 Project-URL: Repository, https://github.com/basicmachines-co/basic-memory
@@ -286,6 +286,92 @@ Basic Memory is built on some key ideas:
 - Simple text patterns can capture rich meaning
 - Local-first doesn't mean feature-poor
 
+## Importing data
+
+Basic memory has cli commands to import data from several formats into Markdown files
+
+### Claude.ai
+
+First, request an export of your data from your Claude account. The data will be emailed to you in several files,
+including
+`conversations.json` and `projects.json`.
+
+Import Claude.ai conversation data
+
+```bash
+ basic-memory import claude conversations conversations.json
+```
+
+The conversations will be turned into Markdown files and placed in the "conversations" folder by default (this can be
+changed with the --folder arg).
+
+Example:
+
+```bash
+Importing chats from conversations.json...writing to .../basic-memory
+  Reading chat data... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+╭────────────────────────────╮
+│ Import complete!           │
+│                            │
+│ Imported 307 conversations │
+│ Containing 7769 messages   │
+╰────────────────────────────╯
+```
+
+Next, you can run the `sync` command to import the data into basic-memory
+
+```bash
+basic-memory sync
+```
+
+You can also import project data from Claude.ai
+
+```bash 
+➜  basic-memory import claude projects
+Importing projects from projects.json...writing to .../basic-memory/projects
+  Reading project data... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+╭────────────────────────────────╮
+│ Import complete!               │
+│                                │
+│ Imported 101 project documents │
+│ Imported 32 prompt templates   │
+╰────────────────────────────────╯
+
+Run 'basic-memory sync' to index the new files.
+```
+
+### Chat Gpt
+
+```bash
+ ➜  basic-memory import chatgpt
+Importing chats from conversations.json...writing to .../basic-memory/conversations
+
+  Reading chat data... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+╭────────────────────────────╮
+│ Import complete!           │
+│                            │
+│ Imported 198 conversations │
+│ Containing 11777 messages  │
+╰────────────────────────────╯
+
+
+```
+
+### Memory json
+
+```bash
+➜  basic-memory import memory-json
+Importing from memory.json...writing to .../basic-memory
+  Reading memory.json... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+  Creating entities...   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+╭──────────────────────╮
+│ Import complete!     │
+│                      │
+│ Created 126 entities │
+│ Added 252 relations  │
+╰──────────────────────╯
+```
+
 ## License
 
 AGPL-3.0

{basic_memory-0.2.21 → basic_memory-0.4.0}/README.md

@@ -253,6 +253,92 @@ Basic Memory is built on some key ideas:
 - Simple text patterns can capture rich meaning
 - Local-first doesn't mean feature-poor
 
+## Importing data
+
+Basic memory has cli commands to import data from several formats into Markdown files
+
+### Claude.ai
+
+First, request an export of your data from your Claude account. The data will be emailed to you in several files,
+including
+`conversations.json` and `projects.json`.
+
+Import Claude.ai conversation data
+
+```bash
+ basic-memory import claude conversations conversations.json
+```
+
+The conversations will be turned into Markdown files and placed in the "conversations" folder by default (this can be
+changed with the --folder arg).
+
+Example:
+
+```bash
+Importing chats from conversations.json...writing to .../basic-memory
+  Reading chat data... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+╭────────────────────────────╮
+│ Import complete!           │
+│                            │
+│ Imported 307 conversations │
+│ Containing 7769 messages   │
+╰────────────────────────────╯
+```
+
+Next, you can run the `sync` command to import the data into basic-memory
+
+```bash
+basic-memory sync
+```
+
+You can also import project data from Claude.ai
+
+```bash 
+➜  basic-memory import claude projects
+Importing projects from projects.json...writing to .../basic-memory/projects
+  Reading project data... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+╭────────────────────────────────╮
+│ Import complete!               │
+│                                │
+│ Imported 101 project documents │
+│ Imported 32 prompt templates   │
+╰────────────────────────────────╯
+
+Run 'basic-memory sync' to index the new files.
+```
+
+### Chat Gpt
+
+```bash
+ ➜  basic-memory import chatgpt
+Importing chats from conversations.json...writing to .../basic-memory/conversations
+
+  Reading chat data... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+╭────────────────────────────╮
+│ Import complete!           │
+│                            │
+│ Imported 198 conversations │
+│ Containing 11777 messages  │
+╰────────────────────────────╯
+
+
+```
+
+### Memory json
+
+```bash
+➜  basic-memory import memory-json
+Importing from memory.json...writing to .../basic-memory
+  Reading memory.json... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+  Creating entities...   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%
+╭──────────────────────╮
+│ Import complete!     │
+│                      │
+│ Created 126 entities │
+│ Added 252 relations  │
+╰──────────────────────╯
+```
+
 ## License
 
 AGPL-3.0

{basic_memory-0.2.21 → basic_memory-0.4.0}/installer/installer.py

@@ -8,6 +8,7 @@ if sys.platform == "darwin":
     import tkinter as tk
     from tkinter import messagebox
 
+
 def ensure_uv_installed():
     """Check if uv is installed, install if not."""
     try:

{basic_memory-0.2.21 → basic_memory-0.4.0}/installer/setup.py

@@ -4,31 +4,24 @@ import sys
 # Build options for all platforms
 build_exe_options = {
     "packages": ["json", "pathlib"],
-    "excludes": [
-        "unittest", 
-        "pydoc",
-        "test"
-    ],
+    "excludes": ["unittest", "pydoc", "test"],
 }
 
 # Platform-specific options
 if sys.platform == "win32":
     base = "Win32GUI"  # Use GUI base for Windows
-    build_exe_options.update({
-        "include_msvcr": True,
-    })
+    build_exe_options.update(
+        {
+            "include_msvcr": True,
+        }
+    )
     target_name = "Basic Memory Installer.exe"
 else:  # darwin
     base = None  # Don't use GUI base for macOS
     target_name = "Basic Memory Installer"
 
 executables = [
-    Executable(
-        script="installer.py",
-        target_name=target_name,
-        base=base,
-        icon="Basic.icns"
-    )
+    Executable(script="installer.py", target_name=target_name, base=base, icon="Basic.icns")
 ]
 
 setup(
@@ -41,7 +34,7 @@ setup(
             "bundle_name": "Basic Memory Installer",
             "iconfile": "Basic.icns",
             "codesign_identity": "-",  # Force ad-hoc signing
-        }
+        },
     },
     executables=executables,
 )

{basic_memory-0.2.21 → basic_memory-0.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "basic-memory"
-version = "0.2.21"
+version = "0.4.0"
 description = "Local-first knowledge management combining Zettelkasten with knowledge graphs"
 readme = "README.md"
 requires-python = ">=3.12.1"
@@ -84,7 +84,9 @@ pythonVersion = "3.12"
 
 
 [tool.semantic_release]
-version_variable = "src/basic_memory/__init__.py:__version__"
+version_variables = [
+    "src/basic_memory/__init__.py:__version__",
+]
 version_toml = [
     "pyproject.toml:project.version",
 ]

{basic_memory-0.2.21 → basic_memory-0.4.0}/src/basic_memory/__init__.py

@@ -1,3 +1,3 @@
 """basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
 
-__version__ = "0.0.1"
+__version__ = "0.4.0"

{basic_memory-0.2.21 → basic_memory-0.4.0}/src/basic_memory/api/app.py

@@ -6,38 +6,17 @@ from fastapi import FastAPI, HTTPException
 from fastapi.exception_handlers import http_exception_handler
 from loguru import logger
 
+import basic_memory
 from basic_memory import db
 from basic_memory.config import config as app_config
 from basic_memory.api.routers import knowledge, search, memory, resource
-from alembic import command
-from alembic.config import Config
-
-from basic_memory.db import DatabaseType
-from basic_memory.repository.search_repository import SearchRepository
-
-
-async def run_migrations():  # pragma: no cover
-    """Run any pending alembic migrations."""
-    logger.info("Running database migrations...")
-    try:
-        config = Config("alembic.ini")
-        command.upgrade(config, "head")
-        logger.info("Migrations completed successfully")
-
-        _, session_maker = await db.get_or_create_db(
-            app_config.database_path, DatabaseType.FILESYSTEM
-        )
-        await SearchRepository(session_maker).init_search_index()
-    except Exception as e:
-        logger.error(f"Error running migrations: {e}")
-        raise
 
 
 @asynccontextmanager
 async def lifespan(app: FastAPI):  # pragma: no cover
     """Lifecycle manager for the FastAPI app."""
-    logger.info("Starting Basic Memory API")
-    await run_migrations()
+    logger.info(f"Starting Basic Memory API {basic_memory.__version__}")
+    await db.run_migrations(app_config)
     yield
     logger.info("Shutting down Basic Memory API")
     await db.shutdown_db()

basic_memory-0.4.0/src/basic_memory/cli/app.py

@@ -0,0 +1,20 @@
+import asyncio
+
+import typer
+
+from basic_memory import db
+from basic_memory.config import config
+from basic_memory.utils import setup_logging
+
+setup_logging(log_file=".basic-memory/basic-memory-cli.log")  # pragma: no cover
+
+asyncio.run(db.run_migrations(config))
+
+app = typer.Typer(name="basic-memory")
+
+import_app = typer.Typer()
+app.add_typer(import_app, name="import")
+
+
+claude_app = typer.Typer()
+import_app.add_typer(claude_app, name="claude")

{basic_memory-0.2.21 → basic_memory-0.4.0}/src/basic_memory/cli/commands/db.py

@@ -22,4 +22,4 @@ def reset(
             from basic_memory.cli.commands.sync import sync
 
             logger.info("Rebuilding search index from filesystem...")
-            asyncio.run(sync())  # pyright: ignore
+            sync(watch=False)  # pyright: ignore

basic_memory-0.4.0/src/basic_memory/cli/commands/import_chatgpt.py

@@ -0,0 +1,255 @@
+"""Import command for ChatGPT conversations."""
+
+import asyncio
+import json
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, Any, List, Annotated, Set, Optional
+
+import typer
+from loguru import logger
+from rich.console import Console
+from rich.panel import Panel
+from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn
+
+from basic_memory.cli.app import import_app
+from basic_memory.config import config
+from basic_memory.markdown import EntityParser, MarkdownProcessor
+from basic_memory.markdown.schemas import EntityMarkdown, EntityFrontmatter
+
+console = Console()
+
+
+def clean_filename(text: str) -> str:
+    """Convert text to safe filename."""
+    clean = "".join(c if c.isalnum() else "-" for c in text.lower()).strip("-")
+    return clean
+
+
+def format_timestamp(ts: float) -> str:
+    """Format Unix timestamp for display."""
+    dt = datetime.fromtimestamp(ts)
+    return dt.strftime("%Y-%m-%d %H:%M:%S")
+
+
+def get_message_content(message: Dict[str, Any]) -> str:
+    """Extract clean message content."""
+    if not message or "content" not in message:
+        return ""  # pragma: no cover
+
+    content = message["content"]
+    if content.get("content_type") == "text":
+        return "\n".join(content.get("parts", []))
+    elif content.get("content_type") == "code":
+        return f"```{content.get('language', '')}\n{content.get('text', '')}\n```"
+    return ""  # pragma: no cover
+
+
+def traverse_messages(
+    mapping: Dict[str, Any], root_id: Optional[str], seen: Set[str]
+) -> List[Dict[str, Any]]:
+    """Traverse message tree and return messages in order."""
+    messages = []
+    node = mapping.get(root_id) if root_id else None
+
+    while node:
+        if node["id"] not in seen and node.get("message"):
+            seen.add(node["id"])
+            messages.append(node["message"])
+
+        # Follow children
+        children = node.get("children", [])
+        for child_id in children:
+            child_msgs = traverse_messages(mapping, child_id, seen)
+            messages.extend(child_msgs)
+
+        break  # Don't follow siblings
+
+    return messages
+
+
+def format_chat_markdown(
+    title: str, mapping: Dict[str, Any], root_id: Optional[str], created_at: float, modified_at: float
+) -> str:
+    """Format chat as clean markdown."""
+
+    # Start with title
+    lines = [f"# {title}\n"]
+
+    # Traverse message tree
+    seen_msgs = set()
+    messages = traverse_messages(mapping, root_id, seen_msgs)
+
+    # Format each message
+    for msg in messages:
+        # Skip hidden messages
+        if msg.get("metadata", {}).get("is_visually_hidden_from_conversation"):
+            continue
+
+        # Get author and timestamp
+        author = msg["author"]["role"].title()
+        ts = format_timestamp(msg["create_time"]) if msg.get("create_time") else ""
+
+        # Add message header
+        lines.append(f"### {author} ({ts})")
+
+        # Add message content
+        content = get_message_content(msg)
+        if content:
+            lines.append(content)
+
+        # Add spacing
+        lines.append("")
+
+    return "\n".join(lines)
+
+
+def format_chat_content(folder: str, conversation: Dict[str, Any]) -> EntityMarkdown:
+    """Convert chat conversation to Basic Memory entity."""
+
+    # Extract timestamps
+    created_at = conversation["create_time"]
+    modified_at = conversation["update_time"]
+
+    root_id = None
+    # Find root message
+    for node_id, node in conversation["mapping"].items():
+        if node.get("parent") is None:
+            root_id = node_id
+            break
+
+    # Generate permalink
+    date_prefix = datetime.fromtimestamp(created_at).strftime("%Y%m%d")
+    clean_title = clean_filename(conversation["title"])
+
+    # Format content
+    content = format_chat_markdown(
+        title=conversation["title"],
+        mapping=conversation["mapping"],
+        root_id=root_id,
+        created_at=created_at,
+        modified_at=modified_at,
+    )
+
+    # Create entity
+    entity = EntityMarkdown(
+        frontmatter=EntityFrontmatter(
+            metadata={
+                "type": "conversation",
+                "title": conversation["title"],
+                "created": format_timestamp(created_at),
+                "modified": format_timestamp(modified_at),
+                "permalink": f"{folder}/{date_prefix}-{clean_title}",
+            }
+        ),
+        content=content,
+    )
+
+    return entity
+
+
+async def process_chatgpt_json(
+    json_path: Path, folder: str, markdown_processor: MarkdownProcessor
+) -> Dict[str, int]:
+    """Import conversations from ChatGPT JSON format."""
+
+    with Progress(
+        SpinnerColumn(),
+        TextColumn("[progress.description]{task.description}"),
+        BarColumn(),
+        TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
+        console=console,
+    ) as progress:
+        read_task = progress.add_task("Reading chat data...", total=None)
+
+        # Read conversations
+        conversations = json.loads(json_path.read_text())
+        progress.update(read_task, total=len(conversations))
+
+        # Process each conversation
+        messages_imported = 0
+        chats_imported = 0
+
+        for chat in conversations:
+            # Convert to entity
+            entity = format_chat_content(folder, chat)
+
+            # Write file
+            file_path = config.home / f"{entity.frontmatter.metadata['permalink']}.md"
+            # logger.info(f"Writing file: {file_path.absolute()}")
+            await markdown_processor.write_file(file_path, entity)
+
+            # Count messages
+            msg_count = sum(
+                1
+                for node in chat["mapping"].values()
+                if node.get("message")
+                and not node.get("message", {})
+                .get("metadata", {})
+                .get("is_visually_hidden_from_conversation")
+            )
+
+            chats_imported += 1
+            messages_imported += msg_count
+            progress.update(read_task, advance=1)
+
+    return {"conversations": chats_imported, "messages": messages_imported}
+
+
+async def get_markdown_processor() -> MarkdownProcessor:
+    """Get MarkdownProcessor instance."""
+    entity_parser = EntityParser(config.home)
+    return MarkdownProcessor(entity_parser)
+
+
+@import_app.command(name="chatgpt", help="Import conversations from ChatGPT JSON export.")
+def import_chatgpt(
+    conversations_json: Annotated[
+        Path, typer.Option(..., help="Path to ChatGPT conversations.json file")
+    ] = Path("conversations.json"),
+    folder: Annotated[
+        str, typer.Option(help="The folder to place the files in.")
+    ] = "conversations",
+):
+    """Import chat conversations from ChatGPT JSON format.
+
+    This command will:
+    1. Read the complex tree structure of messages
+    2. Convert them to linear markdown conversations
+    3. Save as clean, readable markdown files
+
+    After importing, run 'basic-memory sync' to index the new files.
+    """
+
+    try:
+        if conversations_json:
+            if not conversations_json.exists():
+                typer.echo(f"Error: File not found: {conversations_json}", err=True)
+                raise typer.Exit(1)
+
+            # Get markdown processor
+            markdown_processor = asyncio.run(get_markdown_processor())
+
+            # Process the file
+            base_path = config.home / folder
+            console.print(f"\nImporting chats from {conversations_json}...writing to {base_path}")
+            results = asyncio.run(
+                process_chatgpt_json(conversations_json, folder, markdown_processor)
+            )
+
+            # Show results
+            console.print(
+                Panel(
+                    f"[green]Import complete![/green]\n\n"
+                    f"Imported {results['conversations']} conversations\n"
+                    f"Containing {results['messages']} messages",
+                    expand=False,
+                )
+            )
+
+        console.print("\nRun 'basic-memory sync' to index the new files.")
+
+    except Exception as e:
+        logger.error("Import failed")
+        typer.echo(f"Error during import: {e}", err=True)
+        raise typer.Exit(1)