freeplane-and-yaml 0.1.3__tar.gz → 0.3.0__tar.gz

freeplane-and-yaml-0.3.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include README.md
+include LICENSE
+include src/freeplane_and_yaml/schema/mindmap-schema.json
+

freeplane-and-yaml-0.3.0/PKG-INFO

@@ -0,0 +1,99 @@
+Metadata-Version: 2.1
+Name: freeplane-and-yaml
+Version: 0.3.0
+Summary: A tool to convert PDF files to Freeplane mind maps using OpenAI
+Home-page: https://github.com/romilly/freeplane-and-yaml
+Author: Romilly Cocking
+Author-email: romilly.cocking@gmail.com
+License: MIT
+Platform: UNKNOWN
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# PDF to Freeplane Mind Map Converter
+
+A Python tool that converts PDF documents into Freeplane mind maps using OpenAI. This allows you to automatically generate structured mind maps from PDF documents.
+
+You can read about how it got written using AI on [medium](https://medium.com/@romillyc/build-your-own-mind-map-tools-with-ai-b193564f2464?sk=b353aa7d16d6412e4aae8f3eab0ec554).
+That's a _friend link_ so you can read it even if you're not a subscriber.
+
+## Installation
+
+This project requires Python and should be run in a virtual environment:
+
+```bash
+# Create and activate virtual environment in the directory of your choice
+python -m venv venv
+source venv/bin/activate  # On Windows use: venv\Scripts\activate
+pip install freeplane-and-yaml
+```
+
+You'll need to set up your OpenAI API key in a `.env` file or as an environment variable:
+
+```bash
+# Create a .env file in your project directory
+echo "OPENAI_API_KEY=your_api_key_here" > .env
+```
+
+
+## Usage
+
+### Converting PDF Documents to Mind Maps
+
+To convert a PDF document to a Freeplane mind map:
+
+```bash
+# Convert a PDF file to a mind map
+pdf2mindmap path/to/document.pdf output_directory
+
+# Use a specific OpenAI model (default is gpt-4o-mini)
+pdf2mindmap path/to/document.pdf output_directory --model gpt-4-turbo
+
+# Also save the extracted text
+pdf2mindmap path/to/document.pdf output_directory --save-text
+```
+
+The tool performs these steps:
+1. Extracts text from the PDF using PyMuPDF4LLM (preserving structure as markdown)
+2. Processes the text with OpenAI to generate a structured mind map in YAML format
+3. Converts the YAML to a Freeplane mind map (.mm file)
+4. Saves all files in the output directory
+
+This is useful for:
+- Summarizing academic papers
+- Converting product requirements documents (PRDs)
+- Creating structured summaries of technical documentation
+- Organizing research notes or book content
+
+
+The generated `.mm` file can be opened in Freeplane. When you first open the file, Freeplane will show this warning dialog because the file wasn't created by Freeplane itself:
+
+![Freeplane Warning Dialog](images/warning-dialog.png)
+
+This is normal and expected — click OK to load the mind map.
+
+Here's an example of how the output looks:
+
+![Example Mind Map](images/Screenshot%20at%202025-02-12%2010-43-23.png)
+
+## Features
+
+- Converts PDF documents to Freeplane mind maps using OpenAI
+- Uses PyMuPDF4LLM for high-quality text extraction from PDFs
+- Creates hierarchical mind maps with node titles and notes
+- Automatically alternates between right and left positions for top-level nodes
+- Generates unique IDs for each node
+- Produces clean, well-structured YAML as an intermediate format
+- Supports different OpenAI models (default is gpt-4o-mini)
+
+## License
+
+_Apologies to readers from the USA. This README uses UK spelling._
+
+This project is licensed under the MIT Licence — see the [LICENCE](LICENSE) file for details.
+
+

freeplane-and-yaml-0.3.0/README.md

@@ -0,0 +1,81 @@
+# PDF to Freeplane Mind Map Converter
+
+A Python tool that converts PDF documents into Freeplane mind maps using OpenAI. This allows you to automatically generate structured mind maps from PDF documents.
+
+You can read about how it got written using AI on [medium](https://medium.com/@romillyc/build-your-own-mind-map-tools-with-ai-b193564f2464?sk=b353aa7d16d6412e4aae8f3eab0ec554).
+That's a _friend link_ so you can read it even if you're not a subscriber.
+
+## Installation
+
+This project requires Python and should be run in a virtual environment:
+
+```bash
+# Create and activate virtual environment in the directory of your choice
+python -m venv venv
+source venv/bin/activate  # On Windows use: venv\Scripts\activate
+pip install freeplane-and-yaml
+```
+
+You'll need to set up your OpenAI API key in a `.env` file or as an environment variable:
+
+```bash
+# Create a .env file in your project directory
+echo "OPENAI_API_KEY=your_api_key_here" > .env
+```
+
+
+## Usage
+
+### Converting PDF Documents to Mind Maps
+
+To convert a PDF document to a Freeplane mind map:
+
+```bash
+# Convert a PDF file to a mind map
+pdf2mindmap path/to/document.pdf output_directory
+
+# Use a specific OpenAI model (default is gpt-4o-mini)
+pdf2mindmap path/to/document.pdf output_directory --model gpt-4-turbo
+
+# Also save the extracted text
+pdf2mindmap path/to/document.pdf output_directory --save-text
+```
+
+The tool performs these steps:
+1. Extracts text from the PDF using PyMuPDF4LLM (preserving structure as markdown)
+2. Processes the text with OpenAI to generate a structured mind map in YAML format
+3. Converts the YAML to a Freeplane mind map (.mm file)
+4. Saves all files in the output directory
+
+This is useful for:
+- Summarizing academic papers
+- Converting product requirements documents (PRDs)
+- Creating structured summaries of technical documentation
+- Organizing research notes or book content
+
+
+The generated `.mm` file can be opened in Freeplane. When you first open the file, Freeplane will show this warning dialog because the file wasn't created by Freeplane itself:
+
+![Freeplane Warning Dialog](images/warning-dialog.png)
+
+This is normal and expected — click OK to load the mind map.
+
+Here's an example of how the output looks:
+
+![Example Mind Map](images/Screenshot%20at%202025-02-12%2010-43-23.png)
+
+## Features
+
+- Converts PDF documents to Freeplane mind maps using OpenAI
+- Uses PyMuPDF4LLM for high-quality text extraction from PDFs
+- Creates hierarchical mind maps with node titles and notes
+- Automatically alternates between right and left positions for top-level nodes
+- Generates unique IDs for each node
+- Produces clean, well-structured YAML as an intermediate format
+- Supports different OpenAI models (default is gpt-4o-mini)
+
+## License
+
+_Apologies to readers from the USA. This README uses UK spelling._
+
+This project is licensed under the MIT Licence — see the [LICENCE](LICENSE) file for details.

{freeplane-and-yaml-0.1.3 → freeplane-and-yaml-0.3.0}/setup.py

@@ -2,8 +2,8 @@ from setuptools import setup, find_packages
 
 setup(
     name="freeplane-and-yaml",  # Your package name (it must be unique on PyPI)
-    version="0.1.3",  # Initial release version
-    description="A tool to convert YAML files to Freeplane MM format",
+    version="0.3.0",  # Simplified to focus on PDF-to-mindmap with OpenAI
+    description="A tool to convert PDF files to Freeplane mind maps using OpenAI",
     long_description=open("README.md").read(),
     long_description_content_type="text/markdown",  # Use Markdown for PyPI description
     url="https://github.com/romilly/freeplane-and-yaml",  # Replace with your repository URL
@@ -13,8 +13,14 @@ setup(
     packages=find_packages(where="src"),  # Look for packages in src/
     package_dir={"": "src"},  # Root of packages is src/
     include_package_data=True,  # Include non-Python files listed in MANIFEST.in
+    package_data={
+        "freeplane_and_yaml": ["schema/*.json"],  # Include schema files
+    },
     install_requires=[
-        "PyYAML",  # Replace or add dependencies here
+        "PyYAML",
+        "python-dotenv",
+        "openai",
+        "pymupdf4llm",
     ],
     classifiers=[
         "Programming Language :: Python :: 3",
@@ -22,9 +28,9 @@ setup(
         "Operating System :: OS Independent",
     ],
     python_requires=">=3.9", # Specify minimum Python version
-entry_points={
+    entry_points={
         "console_scripts": [
-            "convert=freeplane_and_yaml.cli:main",
+            "pdf2mindmap=freeplane_and_yaml.pdf2mindmap:main",
         ],
     },
 

freeplane-and-yaml-0.3.0/src/freeplane_and_yaml/__init__.py

@@ -0,0 +1,10 @@
+from .convert import convert_yaml_file, converter
+from .llm_adapter import LLMAdapter
+from .text_to_mindmap import TextToMindMap
+
+__all__ = [
+    "convert_yaml_file", 
+    "converter",
+    "LLMAdapter",
+    "TextToMindMap",
+]

{freeplane-and-yaml-0.1.3 → freeplane-and-yaml-0.3.0}/src/freeplane_and_yaml/cli.py

@@ -1,6 +1,6 @@
 import sys
 import os
-from .convert import convert_yaml_file  # Import your conversion logic
+from   freeplane_and_yaml.convert import convert_yaml_file  # Import your conversion logic
 
 
 def main():
@@ -31,10 +31,13 @@ def main():
     base_name = os.path.basename(input_file).replace(".yaml", ".mm")
     output_file = os.path.join(output_dir, base_name)
 
-    # Perform the conversion
+    # Perform the conversio
     try:
         convert_yaml_file(input_file, output_file)
         print(f"Conversion complete: {output_file}")
     except Exception as e:
         print(f"Error during conversion: {e}")
         sys.exit(1)
+
+if __name__ == "__main__":
+    main()

freeplane-and-yaml-0.3.0/src/freeplane_and_yaml/convert_pdf_to_mindmap.py

@@ -0,0 +1,75 @@
+import os
+import sys
+import tempfile
+import pymupdf4llm
+from .text_to_mindmap import TextToMindMap
+from .openai_adapter import OpenAIAdapter
+
+def convert_pdf_to_mindmap(base_name, input_file, mm_output, model, text_output, yaml_output):
+    # Step 1: Extract text from the PDF
+    print(f"Extracting text from {input_file}...")
+    # Ensure the file exists
+    if not os.path.isfile(input_file):
+        raise FileNotFoundError(f"PDF file not found: {input_file}")
+    # Extract text using PyMuPDF4LLM directly
+    extracted_text = pymupdf4llm.to_markdown(input_file)
+    # Save the extracted text if requested
+    if text_output:
+        with open(text_output, 'w', encoding='utf-8') as f:
+            f.write(extracted_text)
+        print(f"Saved extracted text to: {text_output}")
+    # If the text is empty, report an error
+    if not extracted_text.strip():
+        print("Error: No text could be extracted from the PDF", file=sys.stderr)
+        sys.exit(1)
+    # Step 2: Create the OpenAI adapter
+    # Create the adapter with specified model or default
+    adapter = OpenAIAdapter(model=model)
+    # Step 3: Create a temporary text file for the extracted content
+    with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as temp_file:
+        temp_file.write(extracted_text)
+        temp_file_path = temp_file.name
+    try:
+        # Step 4: Process the text with the text-to-mindmap service
+        print("Converting extracted text to mind map format...")
+        service = TextToMindMap(adapter)
+
+        # First try to get the YAML directly to inspect it
+        with open(temp_file_path, 'r', encoding='utf-8') as f:
+            text_content = f.read()
+
+        # Limit the text size to avoid timeouts
+        MAX_CHARS = 40000  # Large documents may cause timeouts
+        if len(text_content) > MAX_CHARS:
+            print(f"Document is large ({len(text_content)} chars). Truncating to {MAX_CHARS} chars to avoid timeouts.")
+            text_content = text_content[:MAX_CHARS] + "\n\n[Content truncated due to length...]"
+
+        try:
+            print(f"Sending document for processing...")
+            yaml_content = adapter.generate_mind_map_yaml(text_content)
+
+            # Save raw output for debugging
+            output_dir = os.path.dirname(mm_output)
+            debug_yaml_path = os.path.join(output_dir, f"{base_name}.raw.yaml")
+            with open(debug_yaml_path, 'w', encoding='utf-8') as f:
+                f.write(yaml_content)
+            print(f"Saved raw YAML for debugging to: {debug_yaml_path}")
+
+            # Now continue with the regular process
+            service.convert_text_file(temp_file_path, yaml_output, mm_output)
+        except Exception as e:
+            print(f"YAML conversion error: {e}")
+            print("Saving debug information and continuing...")
+            # Continue with original code path anyway
+            service.convert_text_file(temp_file_path, yaml_output, mm_output)
+
+        print("\nConversion successful!")
+        print(f"YAML output: {yaml_output}")
+        print(f"Mind map: {mm_output}")
+        if text_output:
+            print(f"Text output: {text_output}")
+
+    finally:
+        # Clean up the temporary file
+        if os.path.exists(temp_file_path):
+            os.unlink(temp_file_path)

freeplane-and-yaml-0.3.0/src/freeplane_and_yaml/llm_adapter.py

@@ -0,0 +1,37 @@
+from abc import ABC, abstractmethod
+from typing import Dict, Any
+import os
+
+# Default schema path - using a path that works both in development and when installed
+# First try the package-relative path (for installed package)
+DEFAULT_SCHEMA_PATH = os.path.join(os.path.dirname(__file__), 'schema', 'mindmap-schema.json')
+
+# If that doesn't exist, try the development path
+if not os.path.exists(DEFAULT_SCHEMA_PATH):
+    DEFAULT_SCHEMA_PATH = os.path.abspath(
+        os.path.join(
+            os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))),
+            "src", "schema", "mindmap-schema.json"
+        )
+    )
+
+
+class LLMAdapter(ABC):
+    """
+    Abstract base class for LLM adapters.
+    This follows the hexagonal architecture pattern to isolate the LLM service from our core logic.
+    """
+    
+    @abstractmethod
+    def generate_mind_map_yaml(self, text: str) -> str:
+        """
+        Generate a mind map in YAML format from the provided text,
+        following the schema defined in DEFAULT_SCHEMA_PATH.
+        
+        Args:
+            text: The text content to convert to a mind map
+            
+        Returns:
+            A string containing the YAML representation of the mind map
+        """
+        pass

freeplane-and-yaml-0.3.0/src/freeplane_and_yaml/openai_adapter.py

@@ -0,0 +1,110 @@
+import os
+from openai import OpenAI
+from typing import Dict, Any
+from dotenv import load_dotenv
+from .llm_adapter import LLMAdapter, DEFAULT_SCHEMA_PATH
+from .yaml_utils import extract_yaml_from_markdown
+
+# Load environment variables from .env file
+load_dotenv()
+
+class OpenAIAdapter(LLMAdapter):
+    """
+    Adapter for OpenAI API.
+    
+    This class provides an implementation of the LLMAdapter for OpenAI models.
+    It makes API calls to OpenAI to generate mind maps from text.
+    """
+    
+    DEFAULT_MODEL = "gpt-4o"  # Default OpenAI model to use
+    
+    def __init__(self, api_key: str = None, model: str = None):
+        """
+        Initialize the adapter with API credentials.
+        
+        Args:
+            api_key: OpenAI API key, defaults to reading from OPENAI_API_KEY environment variable
+            model: OpenAI model to use, defaults to DEFAULT_MODEL
+        """
+        self.api_key = api_key or os.environ.get("OPENAI_API_KEY")
+        if not self.api_key:
+            raise ValueError(
+                "No API key provided. Pass an API key to the constructor or "
+                "set the OPENAI_API_KEY environment variable in the .env file."
+            )
+        
+        self.model = model or self.DEFAULT_MODEL
+        self.client = OpenAI(api_key=self.api_key)
+            
+    def generate_mind_map_yaml(self, text: str) -> str:
+        """
+        Generate a mind map in YAML format using OpenAI.
+        
+        Args:
+            text: The text content to convert to a mind map
+            
+        Returns:
+            A string containing the YAML representation of the mind map
+        """
+        # Read the schema content
+        with open(DEFAULT_SCHEMA_PATH, 'r') as f:
+            schema = f.read()
+            
+        # Make the API call to OpenAI
+        yaml_content = self._call_openai_api(text, schema)
+        
+        return yaml_content
+    
+    def _call_openai_api(self, text: str, schema: str) -> str:
+        """
+        Call the OpenAI API with the text and schema.
+        
+        Args:
+            text: The text content to convert to a mind map
+            schema: The JSON schema content
+            
+        Returns:
+            The generated YAML content
+        """
+        try:
+            # Build system prompt and user prompt
+            system_prompt = "You are an expert at creating structured mind maps in YAML format. Your task is to organize the provided text into a hierarchical structure that follows the schema exactly. Every node must include both a title and a descriptive note providing context or details for that node. Only output valid YAML without any explanations, markdown code blocks, or extra text."
+            
+            user_prompt = f"""I'd like you to summarise a document as a yaml file following a schema.
+
+Please respond with ONLY the YAML content that follows the schema, without any additional text or explanations.
+The YAML should have a root node with an appropriate title and organized children nodes that capture
+the key points and structure of the text. Aim to have one node per 20 lines of input text.
+
+IMPORTANT: Every node (including the root and ALL child nodes) should include both a title AND a note. 
+The note should provide additional context, explanation, or details about that specific node.
+
+Here is the schema to follow:
+
+{schema}
+
+Here is the document to summarize:
+
+{text}"""
+
+            # Call the OpenAI API using the new client interface
+            response = self.client.chat.completions.create(
+                model=self.model,
+                messages=[
+                    {"role": "system", "content": system_prompt},
+                    {"role": "user", "content": user_prompt}
+                ],
+                max_tokens=16000,  # Maximum allowed for this model
+                temperature=0.2,  # Low temperature for more deterministic output
+            )
+            
+            # Extract the response content
+            content = response.choices[0].message.content
+            
+            # Use the utility function to extract YAML from markdown
+            content = extract_yaml_from_markdown(content)
+                
+            return content
+            
+        except Exception as e:
+            raise RuntimeError(f"Error calling OpenAI API: {str(e)}") from e

freeplane-and-yaml-0.3.0/src/freeplane_and_yaml/pdf2mindmap.py

@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+"""
+Convert PDF files to mind maps using PyMuPDF4LLM and AI.
+"""
+
+import os
+import sys
+import argparse
+import tempfile
+import pymupdf4llm
+from dotenv import load_dotenv
+
+from freeplane_and_yaml.convert_pdf_to_mindmap import convert_pdf_to_mindmap
+from .text_to_mindmap import TextToMindMap
+from .openai_adapter import OpenAIAdapter
+
+# Load environment variables
+load_dotenv()
+
+
+def main():
+    """Main entry point for the pdf2mindmap command-line tool."""
+    parser = argparse.ArgumentParser(
+        description="Convert a PDF file to a mind map using PyMuPDF4LLM and OpenAI"
+    )
+    parser.add_argument(
+        "input_file", 
+        help="Path to the input PDF file"
+    )
+    parser.add_argument(
+        "output_dir", 
+        help="Directory to save the output files"
+    )
+    parser.add_argument(
+        "--model", 
+        help="OpenAI model to use (defaults to gpt-4o)"
+    )
+    parser.add_argument(
+        "--save-text",
+        action="store_true",
+        help="Save the extracted text to a file"
+    )
+    
+    args = parser.parse_args()
+    
+    # Validate input file
+    if not os.path.isfile(args.input_file):
+        print(f"Error: Input PDF file '{args.input_file}' does not exist", file=sys.stderr)
+        sys.exit(1)
+        
+    # Make sure it's a PDF file
+    if not args.input_file.lower().endswith('.pdf'):
+        print(f"Error: Input file must be a PDF file: {args.input_file}", file=sys.stderr)
+        sys.exit(1)
+    
+    # Validate output directory
+    if not os.path.isdir(args.output_dir):
+        try:
+            os.makedirs(args.output_dir, exist_ok=True)
+            print(f"Created output directory: {args.output_dir}")
+        except Exception as e:
+            print(f"Error: Could not create output directory '{args.output_dir}': {e}", file=sys.stderr)
+            sys.exit(1)
+    input_file = args.input_file
+    output_dir = args.output_dir
+    save_text = args.save_text
+    # Generate output file paths
+    base_name = os.path.splitext(os.path.basename(input_file))[0]
+    yaml_output = os.path.join(output_dir, f"{base_name}.yaml")
+    mm_output = os.path.join(output_dir, f"{base_name}.mm")
+    text_output = os.path.join(output_dir, f"{base_name}.txt") if save_text else None
+    model = args.model
+    
+    try:
+        convert_pdf_to_mindmap(base_name, input_file, mm_output, model, text_output, yaml_output)
+
+    except Exception as e:
+        print(f"Error during conversion: {e}", file=sys.stderr)
+        sys.exit(1)
+
+
+
+
+if __name__ == "__main__":
+    main()

freeplane-and-yaml-0.3.0/src/freeplane_and_yaml/schema/mindmap-schema.json

@@ -0,0 +1,73 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "title": "Mind Map Schema",
+  "description": "Schema for Freeplane-compatible mind map YAML format",
+  
+  "definitions": {
+    "node": {
+      "type": "object",
+      "required": ["title"],
+      "properties": {
+        "title": {
+          "type": "string",
+          "description": "The display text for the node"
+        },
+        "note": {
+          "type": "string",
+          "description": "Rich text note attached to the node"
+        },
+        "children": {
+          "type": "object",
+          "description": "Child nodes of this node",
+          "patternProperties": {
+            "^[a-zA-Z0-9_]+$": {
+              "$ref": "#/definitions/node"
+            }
+          },
+          "additionalProperties": false
+        }
+      },
+      "additionalProperties": false
+    }
+  },
+
+  "type": "object",
+  "required": ["root"],
+  "properties": {
+    "root": {
+      "allOf": [
+        { "$ref": "#/definitions/node" },
+        { 
+          "required": ["children"],
+          "description": "The root node must have at least one child"
+        }
+      ]
+    }
+  },
+  "additionalProperties": false,
+
+  "examples": [
+    {
+      "root": {
+        "title": "Example Mind Map",
+        "note": "This is the root node",
+        "children": {
+          "topic1": {
+            "title": "First Topic",
+            "note": "Note for first topic",
+            "children": {
+              "subtopic1": {
+                "title": "Subtopic 1",
+                "note": "Note for subtopic"
+              }
+            }
+          },
+          "topic2": {
+            "title": "Second Topic",
+            "note": "Note for second topic"
+          }
+        }
+      }
+    }
+  ]
+}