ain-research 0.1.1__tar.gz

ain_research-0.1.1/.gitignore

@@ -0,0 +1,6 @@
+.env
+*.log
+__pycache__/
+.git/
+.vscode/
+run_publish.bat

ain_research-0.1.1/PKG-INFO

@@ -0,0 +1,75 @@
+Metadata-Version: 2.4
+Name: ain-research
+Version: 0.1.1
+Summary: AIN Infinite Research Daemon. Engineered to sit on top of LLMs to autonomously guide research, provide storage, and retrieval functionality via CLI and MCP.
+Author: That-Tech-Geek
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.9
+Requires-Dist: mcp>=1.2.0
+Description-Content-Type: text/markdown
+
+# AIN Research Daemon Library
+
+Welcome to the **AIN Infinite Research Daemon** (`ain-research`), an open-source Python library built by **That-Tech-Geek**. This library provides the core infrastructure for an Autonomous Intelligence Network (AIN), sitting seamlessly on top of LLMs to guide, automate, and orchestrate open-ended domain research.
+
+## What is it?
+
+`ain-research` is a sophisticated pipeline for automated research ingestion, storage, and retrieval. It allows LLMs and autonomous agents to asynchronously fetch papers from ArXiv, crawl open-source repositories from GitHub, and store them securely into a local SQLite queue before finally syncing them into a hyper-fast, modular knowledge base (the Vault).
+
+Key capabilities include:
+- **Infinite Research Daemon**: Time-boxed, autonomous daemon that fetches domain-specific research using intelligent rate-limiting and rotating APIs.
+- **LLM Steering Integration**: It is engineered to sit on top of LLMs. You can use the CLI or built-in **MCP Server** to dynamically add new search queries, refine categorizations, and prioritize processing.
+- **Atomic Concurrency Control**: Uses OS-level kernel FileLocks to ensure zero data corruption during heavy concurrent reading/writing by different agents.
+- **Model Context Protocol (MCP)**: Directly expose your AIN Second Brain to any MCP-compatible LLM client (like Claude Desktop or other agents), allowing them to queue research, check system health, and compile the wiki on demand.
+
+## Use Cases
+
+1. **Automated Overnight Crawling**
+   Let the daemon run overnight (`ain-daemon`). It crawls ArXiv and GitHub based on your configured categories (e.g., Quant Finance, LLMs) and stores the results securely. By the time you wake up, your Vault is packed with fresh, tagged Markdown files.
+   
+2. **LLM-Guided Research**
+   If you have a primary agent researching "Agentic Frameworks", the agent can use the MCP server (`ain-mcp`) to dynamically add "Agentic Frameworks" to the daytime priority ingestion queue. The daemon will immediately fetch relevant papers and repos, syncing them into the Vault for the LLM to read.
+
+3. **Hyper-fast Retrieval**
+   The core indexing engine (`ain compile`) resolves thousands of nodes into lightweight Maps of Content (MOCs) and exact tag inversions, enabling LLMs to quickly query specific subjects without context-window bloat.
+
+## Installation
+
+```bash
+pip install ain-research
+```
+
+## Quick Start
+
+### 1. Configure Workspace
+By default, the package uses `~/.ain/` as the root workspace. You can override this using the `AIN_WORKSPACE` environment variable.
+
+### 2. Start the Daemon
+```bash
+ain-daemon
+```
+*Runs the overnight crawler. By default, it operates actively between 22:00 and 23:59.*
+
+### 3. Add to Queue via CLI
+```bash
+ain queue add --arxiv 2305.14314
+ain queue add --github karpathy/nanoGPT
+```
+
+### 4. Compile the Knowledge Base
+```bash
+ain compile
+```
+*Generates the Maps of Content (MOCs), Mermaid network graphs, and visualizer data.*
+
+### 5. Launch MCP Server
+To allow LLMs to natively hook into the daemon's research capabilities:
+```bash
+ain-mcp
+```
+*(Configure this in your LLM client's MCP configuration settings).*
+
+## Author
+**That-Tech-Geek**

ain_research-0.1.1/README.md

@@ -0,0 +1,63 @@
+# AIN Research Daemon Library
+
+Welcome to the **AIN Infinite Research Daemon** (`ain-research`), an open-source Python library built by **That-Tech-Geek**. This library provides the core infrastructure for an Autonomous Intelligence Network (AIN), sitting seamlessly on top of LLMs to guide, automate, and orchestrate open-ended domain research.
+
+## What is it?
+
+`ain-research` is a sophisticated pipeline for automated research ingestion, storage, and retrieval. It allows LLMs and autonomous agents to asynchronously fetch papers from ArXiv, crawl open-source repositories from GitHub, and store them securely into a local SQLite queue before finally syncing them into a hyper-fast, modular knowledge base (the Vault).
+
+Key capabilities include:
+- **Infinite Research Daemon**: Time-boxed, autonomous daemon that fetches domain-specific research using intelligent rate-limiting and rotating APIs.
+- **LLM Steering Integration**: It is engineered to sit on top of LLMs. You can use the CLI or built-in **MCP Server** to dynamically add new search queries, refine categorizations, and prioritize processing.
+- **Atomic Concurrency Control**: Uses OS-level kernel FileLocks to ensure zero data corruption during heavy concurrent reading/writing by different agents.
+- **Model Context Protocol (MCP)**: Directly expose your AIN Second Brain to any MCP-compatible LLM client (like Claude Desktop or other agents), allowing them to queue research, check system health, and compile the wiki on demand.
+
+## Use Cases
+
+1. **Automated Overnight Crawling**
+   Let the daemon run overnight (`ain-daemon`). It crawls ArXiv and GitHub based on your configured categories (e.g., Quant Finance, LLMs) and stores the results securely. By the time you wake up, your Vault is packed with fresh, tagged Markdown files.
+   
+2. **LLM-Guided Research**
+   If you have a primary agent researching "Agentic Frameworks", the agent can use the MCP server (`ain-mcp`) to dynamically add "Agentic Frameworks" to the daytime priority ingestion queue. The daemon will immediately fetch relevant papers and repos, syncing them into the Vault for the LLM to read.
+
+3. **Hyper-fast Retrieval**
+   The core indexing engine (`ain compile`) resolves thousands of nodes into lightweight Maps of Content (MOCs) and exact tag inversions, enabling LLMs to quickly query specific subjects without context-window bloat.
+
+## Installation
+
+```bash
+pip install ain-research
+```
+
+## Quick Start
+
+### 1. Configure Workspace
+By default, the package uses `~/.ain/` as the root workspace. You can override this using the `AIN_WORKSPACE` environment variable.
+
+### 2. Start the Daemon
+```bash
+ain-daemon
+```
+*Runs the overnight crawler. By default, it operates actively between 22:00 and 23:59.*
+
+### 3. Add to Queue via CLI
+```bash
+ain queue add --arxiv 2305.14314
+ain queue add --github karpathy/nanoGPT
+```
+
+### 4. Compile the Knowledge Base
+```bash
+ain compile
+```
+*Generates the Maps of Content (MOCs), Mermaid network graphs, and visualizer data.*
+
+### 5. Launch MCP Server
+To allow LLMs to natively hook into the daemon's research capabilities:
+```bash
+ain-mcp
+```
+*(Configure this in your LLM client's MCP configuration settings).*
+
+## Author
+**That-Tech-Geek**

ain_research-0.1.1/ain_research/__init__.py

@@ -0,0 +1,4 @@
+"""
+AIN Research Library
+"""
+__version__ = "0.1.1"

ain_research-0.1.1/ain_research/cli.py

@@ -0,0 +1,42 @@
+import argparse
+import sys
+from ain_research.core import db_manager, orchestrator
+from ain_research.config import ensure_workspace
+
+def main():
+    parser = argparse.ArgumentParser(description="AIN Infinite Research CLI")
+    subparsers = parser.add_subparsers(dest="command")
+
+    # `ain compile`
+    compile_parser = subparsers.add_parser("compile", help="Compile the Vault Index and Tags")
+
+    # `ain queue`
+    queue_parser = subparsers.add_parser("queue", help="Manage the Ingestion Queue")
+    queue_subparsers = queue_parser.add_subparsers(dest="queue_command")
+    
+    queue_add = queue_subparsers.add_parser("add", help="Add to queue")
+    queue_add.add_argument("--arxiv", type=str, help="ArXiv ID")
+    
+    queue_list = queue_subparsers.add_parser("list", help="List pending queue items")
+    
+    args = parser.parse_args()
+    ensure_workspace()
+
+    if args.command == "compile":
+        orchestrator.compile_wiki()
+    elif args.command == "queue":
+        if args.queue_command == "add":
+            if args.arxiv:
+                identifier = f"arxiv_{args.arxiv}"
+                db_manager.enqueue_item("arxiv", identifier, f"ArXiv {args.arxiv}", "", ["arxiv", "queued"], [f"https://arxiv.org/abs/{args.arxiv}"])
+                print(f"[+] Added {args.arxiv} to ingestion queue.")
+        elif args.queue_command == "list":
+            pending = db_manager.get_pending_queue(limit=50)
+            print(f"Pending Items: {len(pending)}")
+            for item in pending:
+                print(f"  - [{item['item_type'].upper()}] {item['identifier']} (retries: {item['retry_count']})")
+    else:
+        parser.print_help()
+
+if __name__ == "__main__":
+    main()

ain_research-0.1.1/ain_research/config.py

@@ -0,0 +1,60 @@
+import os
+import json
+from pathlib import Path
+
+# Base configuration for AIN
+DEFAULT_WORKSPACE = os.path.expanduser("~/.ain")
+AIN_WORKSPACE = os.environ.get("AIN_WORKSPACE", DEFAULT_WORKSPACE)
+
+CRAWL_START_TIME = os.environ.get("CRAWL_START_TIME", "22:00")
+CRAWL_END_TIME = os.environ.get("CRAWL_END_TIME", "23:59:59")
+
+# Sub-directories
+WIKI_DIR = os.path.join(AIN_WORKSPACE, "vault", "wiki")
+DB_DIR = os.path.join(AIN_WORKSPACE, "data")
+
+FOLDERS = {
+    "inbox": os.path.join(WIKI_DIR, "01_Inbox"),
+    "quant_finance": os.path.join(WIKI_DIR, "02_Research", "Quant_Finance"),
+    "machine_learning": os.path.join(WIKI_DIR, "02_Research", "Machine_Learning"),
+    "technology": os.path.join(WIKI_DIR, "02_Research", "Technology"),
+    "mathematics": os.path.join(WIKI_DIR, "02_Research", "Mathematics"),
+    "quantum_physics": os.path.join(WIKI_DIR, "02_Research", "Quantum_Physics"),
+    "economics": os.path.join(WIKI_DIR, "03_Research", "Economics"),
+    "business": os.path.join(WIKI_DIR, "03_Research", "Business"),
+    "vc_sourcing": os.path.join(WIKI_DIR, "04_VC_Sourcing"),
+    "publications": os.path.join(WIKI_DIR, "05_Publications")
+}
+
+def ensure_workspace():
+    """Ensure all required workspace directories exist."""
+    os.makedirs(WIKI_DIR, exist_ok=True)
+    os.makedirs(DB_DIR, exist_ok=True)
+    for folder_path in FOLDERS.values():
+        os.makedirs(folder_path, exist_ok=True)
+
+# Default categories
+DEFAULT_ARXIV_CATEGORIES = {
+    "Finance": {"query": "cat:q-fin.*", "tags": ["finance", "quant", "arxiv"]},
+    "Trading_Algorithms": {"query": "cat:q-fin.TR OR cat:q-fin.CP OR cat:q-fin.MF", "tags": ["trading-algorithms", "algorithmic-trading", "quant", "arxiv"]},
+    "Technology": {"query": "cat:cs.CR OR cat:cs.DC OR cat:cs.NI OR cat:cs.SE OR cat:cs.SY", "tags": ["technology", "computer-science", "arxiv"]},
+    "Machine_Learning": {"query": "cat:cs.LG OR cat:cs.AI OR cat:stat.ML", "tags": ["machine-learning", "ai", "arxiv"]},
+    "Business_Econ": {"query": "cat:econ.* OR cat:q-fin.EC", "tags": ["economics", "business", "arxiv"]},
+    "Mathematics": {"query": "cat:math.*", "tags": ["math", "theory", "arxiv"]},
+    "Quantum_Physics": {"query": "cat:quant-ph", "tags": ["physics", "quantum", "arxiv"]}
+}
+
+def get_arxiv_categories():
+    config_file = os.path.join(AIN_WORKSPACE, "arxiv_categories.json")
+    if os.path.exists(config_file):
+        try:
+            with open(config_file, "r") as f:
+                return json.load(f)
+        except:
+            pass
+    return DEFAULT_ARXIV_CATEGORIES
+
+def save_arxiv_categories(cats):
+    os.makedirs(AIN_WORKSPACE, exist_ok=True)
+    with open(os.path.join(AIN_WORKSPACE, "arxiv_categories.json"), "w") as f:
+        json.dump(cats, f, indent=4)

ain_research-0.1.1/ain_research/core/__init__.py

@@ -0,0 +1 @@
+# Core functionalities

ain_research-0.1.1/ain_research/core/db_manager.py

@@ -0,0 +1,246 @@
+import os
+import sys
+import time
+import json
+import sqlite3
+import traceback
+from datetime import datetime
+
+from ain_research.config import DB_DIR, WIKI_DIR
+
+VAULT_DIR = os.path.dirname(WIKI_DIR)
+LOCK_FILE = os.path.join(VAULT_DIR, "ain_vault.lock")
+DB_FILE = os.path.join(DB_DIR, "ain_system.db")
+
+# Ensure vault and db directories exist
+os.makedirs(VAULT_DIR, exist_ok=True)
+os.makedirs(DB_DIR, exist_ok=True)
+
+# --- 1. OS-Level Kernel Atomic File Lock ---
+class FileLock:
+    """
+    Highly robust, OS-level atomic file lock context manager.
+    Uses standard library os.open with O_CREAT and O_EXCL to achieve atomic lock acquisition.
+    """
+    def __init__(self, lock_file_path=LOCK_FILE, timeout=15):
+        self.lock_file_path = lock_file_path
+        self.timeout = timeout
+        self.has_lock = False
+
+    def acquire(self):
+        start_time = time.time()
+        while time.time() - start_time < self.timeout:
+            try:
+                fd = os.open(self.lock_file_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
+                os.close(fd)
+                self.has_lock = True
+                return True
+            except FileExistsError:
+                time.sleep(0.05)
+            except Exception as e:
+                print(f"[!] Warning: Lock acquisition encountered unexpected error: {e}", file=sys.stderr)
+                time.sleep(0.05)
+        return False
+
+    def release(self):
+        if self.has_lock:
+            try:
+                os.remove(self.lock_file_path)
+            except Exception:
+                pass
+            self.has_lock = False
+
+    def __enter__(self):
+        if not self.acquire():
+            print(f"[!] Lock acquisition timed out after {self.timeout} seconds for {self.lock_file_path}. Proceeding under unsafe fallback mode.", file=sys.stderr)
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.release()
+
+# --- 2. SQLite Database Queue & Error Manager ---
+def get_db_connection():
+    """Returns an active SQLite database connection with a high timeout to handle locks gracefully."""
+    conn = sqlite3.connect(DB_FILE, timeout=10.0)
+    conn.row_factory = sqlite3.Row
+    return conn
+
+def init_db():
+    """Initializes SQLite queue and error monitoring tables."""
+    conn = get_db_connection()
+    cursor = conn.cursor()
+    
+    cursor.execute("""
+    CREATE TABLE IF NOT EXISTS ingestion_queue (
+        id INTEGER PRIMARY KEY AUTOINCREMENT,
+        item_type TEXT NOT NULL,
+        identifier TEXT UNIQUE NOT NULL,
+        title TEXT NOT NULL,
+        content TEXT NOT NULL,
+        tags TEXT NOT NULL,
+        sources TEXT NOT NULL,
+        category TEXT,
+        status TEXT DEFAULT 'pending',
+        retry_count INTEGER DEFAULT 0,
+        error_log TEXT,
+        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+        processed_at TIMESTAMP
+    )
+    """)
+    
+    cursor.execute("""
+    CREATE TABLE IF NOT EXISTS system_errors (
+        id INTEGER PRIMARY KEY AUTOINCREMENT,
+        component TEXT NOT NULL,
+        error_message TEXT NOT NULL,
+        traceback TEXT NOT NULL,
+        timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+    )
+    """)
+
+    conn.commit()
+    conn.close()
+
+
+def enqueue_item(item_type, identifier, title, content, tags, sources, category=None):
+    """
+    Safely transactionally enqueues a fetched research item.
+    Returns True if successfully enqueued, False if it was already processed or queued.
+    """
+    init_db()
+    conn = get_db_connection()
+    cursor = conn.cursor()
+    
+    tags_json = json.dumps(tags)
+    sources_json = json.dumps(sources)
+    
+    try:
+        cursor.execute("""
+        INSERT INTO ingestion_queue (item_type, identifier, title, content, tags, sources, category, status)
+        VALUES (?, ?, ?, ?, ?, ?, ?, 'pending')
+        """, (item_type, identifier, title, content, tags_json, sources_json, category))
+        conn.commit()
+        return True
+    except sqlite3.IntegrityError:
+        return False
+    except Exception as e:
+        log_system_error("QueueManager", f"Failed to enqueue {identifier}: {e}", traceback.format_exc())
+        return False
+    finally:
+        conn.close()
+
+def log_system_error(component, error_message, traceback_str=None):
+    """Logs a system or crawl failure to the SQLite error database."""
+    init_db()
+    try:
+        conn = get_db_connection()
+        cursor = conn.cursor()
+        tb = traceback_str if traceback_str else "".join(traceback.format_stack())
+        cursor.execute("""
+        INSERT INTO system_errors (component, error_message, traceback)
+        VALUES (?, ?, ?)
+        """, (component, error_message, tb))
+        conn.commit()
+        conn.close()
+        print(f"[!] Database logged error inside component '{component}': {error_message}", file=sys.stderr)
+    except Exception as e:
+        print(f"[!!] Critical Error writing to error database: {e}", file=sys.stderr)
+
+def get_pending_queue(limit=100):
+    """Retrieves list of pending queue items for processing."""
+    init_db()
+    conn = get_db_connection()
+    cursor = conn.cursor()
+    cursor.execute("""
+    SELECT id, item_type, identifier, title, content, tags, sources, category, retry_count
+    FROM ingestion_queue
+    WHERE status = 'pending' AND retry_count < 3
+    ORDER BY created_at ASC
+    LIMIT ?
+    """, (limit,))
+    rows = cursor.fetchall()
+    conn.close()
+    
+    items = []
+    for r in rows:
+        items.append({
+            "id": r["id"],
+            "item_type": r["item_type"],
+            "identifier": r["identifier"],
+            "title": r["title"],
+            "content": r["content"],
+            "tags": json.loads(r["tags"]),
+            "sources": json.loads(r["sources"]),
+            "category": r["category"],
+            "retry_count": r["retry_count"]
+        })
+    return items
+
+def mark_item_processed(item_id):
+    """Marks a queue item as successfully processed."""
+    conn = get_db_connection()
+    cursor = conn.cursor()
+    cursor.execute("""
+    UPDATE ingestion_queue
+    SET status = 'processed', processed_at = CURRENT_TIMESTAMP
+    WHERE id = ?
+    """, (item_id,))
+    conn.commit()
+    conn.close()
+
+def mark_item_failed(item_id, error_msg):
+    """Increments retry counts and logs failure details on a queue item."""
+    conn = get_db_connection()
+    cursor = conn.cursor()
+    cursor.execute("""
+    UPDATE ingestion_queue
+    SET retry_count = retry_count + 1, error_log = ?
+    WHERE id = ?
+    """, (error_msg, item_id))
+    
+    cursor.execute("SELECT retry_count, identifier, item_type FROM ingestion_queue WHERE id = ?", (item_id,))
+    row = cursor.fetchone()
+    dead_letter = False
+    dead_letter_info = None
+    if row and row["retry_count"] >= 3:
+        cursor.execute("UPDATE ingestion_queue SET status = 'failed' WHERE id = ?", (item_id,))
+        dead_letter = True
+        dead_letter_info = (row['item_type'], row['identifier'])
+        
+    conn.commit()
+    conn.close()
+    
+    if dead_letter and dead_letter_info:
+        log_system_error(
+            f"DeadLetterQueue_{dead_letter_info[0]}",
+            f"Ingestion failed permanently for {dead_letter_info[1]} after 3 attempts. Error: {error_msg}"
+        )
+
+def get_system_metrics():
+    """Gathers high-signal health metrics for the ain status CLI."""
+    init_db()
+    conn = get_db_connection()
+    cursor = conn.cursor()
+    
+    metrics = {}
+    try:
+        cursor.execute("SELECT status, count(*) as count FROM ingestion_queue GROUP BY status")
+        q_stats = {row["status"]: row["count"] for row in cursor.fetchall()}
+        metrics["queue_pending"] = q_stats.get("pending", 0)
+        metrics["queue_processed"] = q_stats.get("processed", 0)
+        metrics["queue_failed"] = q_stats.get("failed", 0)
+        
+        cursor.execute("SELECT count(*) as count FROM system_errors")
+        metrics["total_errors"] = cursor.fetchone()["count"]
+        
+        cursor.execute("SELECT component, error_message, timestamp FROM system_errors ORDER BY timestamp DESC LIMIT 5")
+        metrics["recent_errors"] = [dict(row) for row in cursor.fetchall()]
+        
+    except Exception as e:
+        metrics["db_error"] = str(e)
+    finally:
+        conn.close()
+        
+    return metrics
+
+init_db()

ain_research-0.1.1/ain_research/core/orchestrator.py

@@ -0,0 +1,68 @@
+import os
+import re
+import json
+from datetime import datetime
+from ain_research.config import WIKI_DIR, FOLDERS
+from ain_research.core import db_manager
+
+def clean_title_to_filename(title):
+    clean = re.sub(r"[^\w\s\-]", "", title)
+    clean = re.sub(r"[\s\-]+", "_", clean)
+    return clean.strip("_")
+
+def compile_wiki():
+    print("[*] Launching AIN Wiki Compiler...")
+    
+    with db_manager.FileLock():
+        all_pages = {}
+        
+        # Step 1: Scan all files
+        for root, _, files in os.walk(WIKI_DIR):
+            for file in files:
+                if file.endswith(".md") and not file.endswith("_MOC.md") and file != "INDEX.md":
+                    file_path = os.path.join(root, file)
+                    slug = os.path.splitext(file)[0]
+                    
+                    try:
+                        with open(file_path, "r", encoding="utf-8", errors="ignore") as f:
+                            content = f.read()
+                    except Exception as e:
+                        continue
+                        
+                    title_match = re.search(r"^#\s+(.+)$", content, re.MULTILINE)
+                    title = title_match.group(1).strip() if title_match else slug.replace("_", " ")
+                    
+                    tags = []
+                    tags_match = re.search(r"tags:\s*\[(.*?)\]", content)
+                    if tags_match:
+                        tags = [t.strip().strip('"').strip("'") for t in tags_match.group(1).split(",") if t.strip()]
+                    
+                    all_pages[slug] = {
+                        "title": title,
+                        "slug": slug,
+                        "file_path": file_path,
+                        "tags": tags
+                    }
+        
+        # Step 2: Build tag index
+        tag_index = {}
+        for slug, info in all_pages.items():
+            for tag in info["tags"]:
+                if tag not in tag_index:
+                    tag_index[tag] = []
+                tag_index[tag].append(slug)
+                
+        tag_index_path = os.path.join(os.path.dirname(WIKI_DIR), "tag_index.json")
+        with open(tag_index_path, "w", encoding="utf-8") as f:
+            json.dump(tag_index, f, indent=2)
+            
+        # Step 3: Build simple INDEX.md
+        index_content = f"# AIN Second Brain Index\n\nGenerated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n## Recent Nodes\n"
+        
+        for slug, info in list(all_pages.items())[:50]:
+            index_content += f"- [[{slug}]] - {info['title']}\n"
+            
+        with open(os.path.join(WIKI_DIR, "INDEX.md"), "w", encoding="utf-8") as f:
+            f.write(index_content)
+            
+        print(f"[+] Compiled {len(all_pages)} research pages into the index.")

ain_research-0.1.1/ain_research/core/organize.py

@@ -0,0 +1,38 @@
+import os
+from ain_research.config import FOLDERS
+from ain_research.core import db_manager
+
+def organize_inbox():
+    """Reads pending items from SQLite queue and writes them to the Inbox."""
+    print("[*] Organizing Inbox from database queue...")
+    pending = db_manager.get_pending_queue(limit=500)
+    
+    if not pending:
+        print("    -> No pending items to organize.")
+        return
+
+    processed = 0
+    inbox_dir = FOLDERS["inbox"]
+    
+    with db_manager.FileLock():
+        for item in pending:
+            identifier = item["identifier"]
+            content = item["content"]
+            
+            # Sanitize filename
+            safe_filename = "".join([c if c.isalnum() or c in ['-', '_'] else "_" for c in identifier]) + ".md"
+            file_path = os.path.join(inbox_dir, safe_filename)
+            
+            try:
+                with open(file_path, "w", encoding="utf-8") as f:
+                    f.write(content)
+                db_manager.mark_item_processed(item["id"])
+                processed += 1
+            except Exception as e:
+                print(f"[!] Error writing {identifier} to disk: {e}")
+                db_manager.mark_item_failed(item["id"], str(e))
+                
+    print(f"[+] Inbox organization complete. Wrote {processed} new nodes.")
+
+if __name__ == "__main__":
+    organize_inbox()

ain_research-0.1.1/ain_research/daemon.py

@@ -0,0 +1,204 @@
+import os
+import sys
+import time
+import json
+import urllib.request
+import urllib.parse
+import urllib.error
+import xml.etree.ElementTree as ET
+import argparse
+from datetime import datetime, timedelta
+import traceback
+
+from ain_research.config import (
+    AIN_WORKSPACE,
+    CRAWL_START_TIME,
+    CRAWL_END_TIME,
+    get_arxiv_categories,
+    save_arxiv_categories,
+    ensure_workspace
+)
+from ain_research.core import db_manager
+
+# Reconfigure standard output encoding
+if hasattr(sys.stdout, 'reconfigure'):
+    sys.stdout.reconfigure(encoding='utf-8')
+
+# State Files
+ARXIV_STATE_FILE = os.path.join(AIN_WORKSPACE, "daemon_state.json")
+
+ARXIV_CATEGORIES = get_arxiv_categories()
+EXISTING_RESEARCH_TITLES = set()
+
+def is_in_active_window(start_time_str="22:00", end_time_str="23:59:59"):
+    now = datetime.now().time()
+    try:
+        start_time = datetime.strptime(start_time_str, "%H:%M").time()
+    except ValueError:
+        start_time = datetime.strptime(start_time_str, "%H:%M:%S").time()
+    try:
+        end_time = datetime.strptime(end_time_str, "%H:%M").time()
+    except ValueError:
+        end_time = datetime.strptime(end_time_str, "%H:%M:%S").time()
+    
+    if start_time <= end_time:
+        return start_time <= now <= end_time
+    else:
+        return now >= start_time or now <= end_time
+
+def load_arxiv_state():
+    if os.path.exists(ARXIV_STATE_FILE):
+        try:
+            with open(ARXIV_STATE_FILE, "r") as f:
+                state = json.load(f)
+                if state and isinstance(next(iter(state.values())), int):
+                    state = {cat: {"latest_seen_id": "", "last_run": ""} for cat in ARXIV_CATEGORIES.keys()}
+                return state
+        except:
+            pass
+    return {cat: {"latest_seen_id": "", "last_run": ""} for cat in ARXIV_CATEGORIES.keys()}
+
+def save_arxiv_state(state):
+    os.makedirs(AIN_WORKSPACE, exist_ok=True)
+    with open(ARXIV_STATE_FILE, "w") as f:
+        json.dump(state, f, indent=4)
+
+def clean_filename(title):
+    clean = "".join([c if c.isalnum() or c.isspace() else "_" for c in title])
+    clean = clean.replace(" ", "_")
+    clean = "_".join(filter(None, clean.split("_")))
+    return clean[:100]
+
+def trigger_sync(is_shutdown=False):
+    if is_shutdown:
+        print("\n[!] Graceful Shutdown Triggered! Running final AIN Sync...")
+    else:
+        print("[*] Triggering AIN Sync pipeline...")
+        
+    try:
+        from ain_research.core import organize, orchestrator
+        organize.organize_inbox()
+        orchestrator.compile_wiki()
+        print("[+] AIN Sync completed successfully.")
+    except Exception as e:
+        print(f"[!] AIN Sync encountered an error: {e}")
+        traceback.print_exc()
+
+def fetch_arxiv_papers(category, query_data, state_entry, batch_size=20):
+    print(f"[{datetime.now().strftime('%H:%M:%S')}] Daemon: Fetching ArXiv '{category}'...")
+    query = query_data["query"]
+    tags = query_data["tags"]
+    latest_seen_id = state_entry.get("latest_seen_id", "")
+    start_offset = 0
+    max_pages = 2
+    processed = 0
+    new_latest_id = None
+    stop_fetching = False
+    
+    for page in range(max_pages):
+        url = f"https://export.arxiv.org/api/query?search_query={urllib.parse.quote(query)}&start={start_offset}&max_results={batch_size}&sortBy=submittedDate&sortOrder=descending"
+        req = urllib.request.Request(url, headers={'User-Agent': 'AIN-Daemon-Bot'})
+        
+        try:
+            response = urllib.request.urlopen(req)
+            xml_data = response.read()
+            root = ET.fromstring(xml_data)
+            ns = {'atom': 'http://www.w3.org/2005/Atom'}
+            entries = root.findall('atom:entry', ns)
+            
+            if not entries:
+                break
+                
+            if page == 0 and len(entries) > 0:
+                first_id = entries[0].find('atom:id', ns)
+                if first_id is not None and first_id.text:
+                    new_latest_id = first_id.text.strip()
+            
+            for entry in entries:
+                id_elem = entry.find('atom:id', ns)
+                link = id_elem.text.strip() if id_elem is not None else ""
+                
+                if latest_seen_id and link == latest_seen_id:
+                    stop_fetching = True
+                    break
+                
+                title_elem = entry.find('atom:title', ns)
+                title = title_elem.text.replace('\\n', ' ').strip() if title_elem is not None else "Untitled"
+                summary_elem = entry.find('atom:summary', ns)
+                summary = summary_elem.text.replace('\\n', ' ').strip() if summary_elem is not None else "No abstract."
+                
+                safe_title = clean_filename(title)
+                date_str = datetime.now().strftime("%Y-%m-%d")
+                
+                md_content = f"""---
+title: "{title.replace('"', "'")}"
+date: "{date_str}"
+tags: {json.dumps(tags)}
+sources: ["{link}"]
+---
+# {title}
+## Source: {link}
+## Abstract
+{summary}
+"""
+                identifier = f"Daemon_ArXiv_{category}_{safe_title}"
+                success = db_manager.enqueue_item("arxiv", identifier, title, md_content, tags, [link], category)
+                if success:
+                    processed += 1
+                
+            if stop_fetching or len(entries) < batch_size:
+                break
+                
+            start_offset += batch_size
+            time.sleep(2)
+            
+        except Exception as e:
+            print(f"    -> Error fetching ArXiv: {e}")
+            break
+            
+    if new_latest_id:
+        state_entry["latest_seen_id"] = new_latest_id
+        state_entry["last_run"] = datetime.now().strftime("%Y-%m-%d")
+        
+    return processed > 0
+
+def main():
+    parser = argparse.ArgumentParser(description="AIN Research Ingestion Daemon")
+    parser.add_argument("--force", action="store_true", help="Force run outside active window")
+    parser.add_argument("--start-time", type=str, default=CRAWL_START_TIME)
+    parser.add_argument("--end-time", type=str, default=CRAWL_END_TIME)
+    args = parser.parse_args()
+
+    ensure_workspace()
+    arxiv_state = load_arxiv_state()
+    arxiv_keys = list(ARXIV_CATEGORIES.keys())
+    arxiv_idx = 0
+    
+    while True:
+        try:
+            if not args.force and not is_in_active_window(args.start_time, args.end_time):
+                print(f"[*] Current time outside active window ({args.start_time}-{args.end_time}). Exiting gracefully...")
+                trigger_sync(is_shutdown=True)
+                sys.exit(0)
+                
+            if arxiv_keys:
+                cat_name = arxiv_keys[arxiv_idx]
+                cat_data = ARXIV_CATEGORIES[cat_name]
+                state_entry = arxiv_state.get(cat_name, {"latest_seen_id": "", "last_run": ""})
+                
+                fetch_arxiv_papers(cat_name, cat_data, state_entry)
+                arxiv_state[cat_name] = state_entry
+                save_arxiv_state(arxiv_state)
+                arxiv_idx = (arxiv_idx + 1) % len(arxiv_keys)
+                
+            time.sleep(5)
+            
+        except KeyboardInterrupt:
+            trigger_sync(is_shutdown=True)
+            break
+        except Exception as e:
+            print(f"[!] Daemon Error: {e}")
+            time.sleep(10)
+
+if __name__ == "__main__":
+    main()

ain_research-0.1.1/ain_research/mcp_server.py

@@ -0,0 +1,77 @@
+from mcp.server.fastmcp import FastMCP
+from ain_research.core import db_manager, orchestrator
+from ain_research.config import (
+    ensure_workspace,
+    get_arxiv_categories,
+    save_arxiv_categories
+)
+
+# Initialize FastMCP Server
+mcp = FastMCP("AIN-Research-Daemon")
+
+@mcp.tool()
+def queue_arxiv(arxiv_id: str) -> str:
+    """Queue an ArXiv paper for autonomous research ingestion."""
+    ensure_workspace()
+    identifier = f"arxiv_{arxiv_id}"
+    success = db_manager.enqueue_item(
+        "arxiv", 
+        identifier, 
+        f"ArXiv {arxiv_id}", 
+        "", 
+        ["arxiv", "queued", "llm-guided"], 
+        [f"https://arxiv.org/abs/{arxiv_id}"]
+    )
+    if success:
+        return f"Successfully queued ArXiv paper {arxiv_id}."
+    return f"Failed to queue {arxiv_id} (may already exist)."
+
+
+
+@mcp.tool()
+def get_queue_status() -> str:
+    """Check the status of the research ingestion queue."""
+    ensure_workspace()
+    pending = db_manager.get_pending_queue(limit=100)
+    metrics = db_manager.get_system_metrics()
+    
+    status = f"Queue Metrics:\nPending: {metrics.get('queue_pending', 0)}\nProcessed: {metrics.get('queue_processed', 0)}\nFailed: {metrics.get('queue_failed', 0)}\n\n"
+    if pending:
+        status += "Next 5 items to process:\n"
+        for item in pending[:5]:
+            status += f"- [{item['item_type'].upper()}] {item['identifier']}\n"
+    return status
+
+@mcp.tool()
+def trigger_compile() -> str:
+    """Trigger a compilation of the AIN knowledge base to update Maps of Content."""
+    ensure_workspace()
+    try:
+        orchestrator.compile_wiki()
+        return "Compilation completed successfully."
+    except Exception as e:
+        return f"Compilation failed: {e}"
+
+@mcp.tool()
+def add_arxiv_category(category_name: str, query: str, tags: list[str]) -> str:
+    """
+    Add a new research category for the automated daemon to crawl on ArXiv.
+    Args:
+        category_name: Name of the category (e.g., 'Agentic_AI').
+        query: ArXiv search query (e.g., 'cat:cs.AI OR cat:cs.LG').
+        tags: List of tags to apply to downloaded papers.
+    """
+    ensure_workspace()
+    cats = get_arxiv_categories()
+    cats[category_name] = {"query": query, "tags": tags}
+    save_arxiv_categories(cats)
+    return f"Added new ArXiv research trajectory: {category_name}"
+
+
+
+def main():
+    ensure_workspace()
+    mcp.run()
+
+if __name__ == "__main__":
+    main()

ain_research-0.1.1/pyproject.toml

@@ -0,0 +1,26 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "ain-research"
+version = "0.1.1"
+description = "AIN Infinite Research Daemon. Engineered to sit on top of LLMs to autonomously guide research, provide storage, and retrieval functionality via CLI and MCP."
+readme = "README.md"
+requires-python = ">=3.9"
+authors = [
+    { name = "That-Tech-Geek" }
+]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+dependencies = [
+    "mcp>=1.2.0",
+]
+
+[project.scripts]
+ain = "ain_research.cli:main"
+ain-daemon = "ain_research.daemon:main"
+ain-mcp = "ain_research.mcp_server:main"