achem 1.0.0__tar.gz

achem-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ACHEM Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

achem-1.0.0/PKG-INFO

@@ -0,0 +1,253 @@
+Metadata-Version: 2.4
+Name: achem
+Version: 1.0.0
+Summary: Deep Web Research Tool - Aggregates 30+ sources, scrapes content, generates AI summaries
+Author: ACHEM Contributors
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/achem/achem
+Project-URL: Documentation, https://github.com/achem/achem#readme
+Project-URL: Repository, https://github.com/achem/achem
+Project-URL: Issues, https://github.com/achem/achem/issues
+Keywords: research,deep-web,web-scraping,summarization,ai,cli,tool
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Text Processing :: General
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: wikipedia-api>=0.5.4
+Requires-Dist: rich>=13.0.0
+Requires-Dist: psutil>=5.9.0
+Requires-Dist: prompt_toolkit>=3.0.0
+Requires-Dist: pyfiglet>=0.8.0
+Requires-Dist: openai>=1.0.0
+Requires-Dist: ddgs>=3.0.0
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: requests>=2.31.0
+Provides-Extra: arabic
+Requires-Dist: arabic-reshaper>=3.0.0; extra == "arabic"
+Requires-Dist: python-bidi>=0.14.0; extra == "arabic"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Dynamic: license-file
+
+# ACHEM - Deep Web Research Tool
+
+![ACHEM Banner](https://img.shields.io/badge/ACHEM-v1.0.0-blue?style=for-the-badge)
+
+> **ACHEM** (Arabic: آشم) is a powerful deep web research tool that aggregates information from 30+ sources, scrapes full content from top results, and generates concise summaries using AI.
+
+## Features
+
+- **Deep Web Research**: Gathers results from 30+ sources via DuckDuckGo
+- **Web Scraping**: Extracts full content from top 3 most relevant links
+- **Two-Pass Search**: Prioritizes technical content (StackOverflow, GitHub, forums)
+- **AI Summarization**: Uses Hugging Face Inference Providers (free tier)
+- **Syntax Highlighting**: Color-coded output for easy scanning
+- **SQLite Cache**: Instant recall for repeated searches
+- **Export**: Save summaries to Markdown files
+- **Multi-language**: Supports English, French, and Arabic
+
+## Screenshots
+
+```
+╔══════════════════════════════════════════════════════════════════╗
+║                    ACHEM - Deep Web Research                      ║
+╚══════════════════════════════════════════════════════════════════╝
+
+🔍 Deep Research: how to learn python
+==================================================
+PASS 1: Gathering 30 sources from DuckDuckGo...
+✓ Found 30 sources
+PASS 2: Scraped full content from top 3 links
+→ Analyzing 35 total sources...
+→ Generating deep summary...
+
+╭──────────────────────────────────────────────────────────────────╮
+│ UNIFIED RESEARCH SUMMARY                                          │
+├──────────────────────────────────────────────────────────────────┤
+│ 1. Start with the official Python tutorial:                     │
+│    - Visit docs.python.org/3/tutorial                           │
+│                                                                  │
+│ 2. Use free online tutorials:                                   │
+│    - LearnPython.org, pythonbasics.org                         │
+╰──────────────────────────────────────────────────────────────────╯
+```
+
+## Installation
+
+### Prerequisites
+
+- Python 3.10 or higher
+- pip package manager
+
+### Quick Install (PyPI)
+
+```bash
+pip install achem
+```
+
+### Or Install from Source
+
+1. **Clone the repository**
+```bash
+git clone https://github.com/achem/achem.git
+cd achem
+```
+
+2. **Install in editable mode**
+```bash
+pip install -e .
+```
+
+3. **Configure API keys**
+```bash
+cp .env.example .env
+```
+
+Then edit `.env` and add your Hugging Face API token:
+```env
+HF_API_KEY=hf_your_token_here
+HF_MODEL=Qwen/Qwen2.5-7B-Instruct
+```
+
+### Getting a Hugging Face API Token
+
+1. Go to [Hugging Face](https://huggingface.co/)
+2. Create an account (free)
+3. Go to Settings → Access Tokens
+4. Create a new token with "Read" permissions
+5. Copy the token to your `.env` file
+
+## Usage
+
+### Interactive Mode
+
+```bash
+python src/main.py
+```
+
+### Command Line Mode
+
+```bash
+python src/main.py "your search query"
+```
+
+### Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-l, --limit` | Wikipedia results per query | 10 |
+| `--lang` | Language (en/fr/ar/auto) | auto |
+| `--ddg-limit` | DuckDuckGo results | 30 |
+| `--min-relevance` | Minimum relevance % | 0 |
+| `--no-cache` | Skip cache | False |
+| `--no-wikipedia` | Skip Wikipedia | False |
+| `--clear-cache` | Clear SQLite cache | False |
+
+### Commands (Interactive Mode)
+
+| Command | Description |
+|---------|-------------|
+| `clear` / `cls` | Clear screen |
+| `export` / `save` | Export last summary |
+| `help` / `?` | Show help |
+| `version` / `v` | Show version |
+| `exit` / `quit` / `q` | Exit program |
+
+## Project Structure
+
+```
+ACHEM/
+├── src/
+│   └── achem/              # Main package
+│       ├── __init__.py
+│       ├── main.py        # Entry point
+│       ├── commands.py    # Command handler
+│       ├── config_manager.py    # Config loader
+│       ├── duckduckgo_client.py # DDG search
+│       ├── export_manager.py    # Export to Documents/ACHEM/
+│       ├── huggingface_summarizer.py # AI summarization
+│       ├── output_formatter.py # Terminal UI
+│       ├── search_router.py    # Source priority
+│       ├── sqlite_cache.py     # SQLite cache
+│       ├── spell_checker.py    # Typo correction
+│       ├── text_analyzer.py    # TF-IDF analysis
+│       ├── user_input.py       # Input handler
+│       ├── web_scraper.py      # BeautifulSoup scraper
+│       └── wikipedia_client.py  # Wikipedia API
+├── .env.example            # Config template
+├── .gitignore
+├── LICENSE
+├── README.md
+└── pyproject.toml         # Package metadata
+```
+
+## How It Works
+
+### Two-Pass Search System
+
+```
+┌─────────────────────────────────────────────────────┐
+│ PASS 1: DuckDuckGo Search (30 results)              │
+│ • Prioritizes technical sites                        │
+│ • Filters out cookie/login/consent pages            │
+│ • Ranks by domain authority                         │
+├─────────────────────────────────────────────────────┤
+│ PASS 2: Web Scraping (Top 3)                        │
+│ • BeautifulSoup extracts full article text           │
+│ • Removes navigation/footer/scripts                 │
+│ • Combines up to 10,000 chars per article           │
+├─────────────────────────────────────────────────────┤
+│ PASS 3: AI Summarization                            │
+│ • Neutral technical prompt                           │
+│ • No ethical warnings or opinions                   │
+│ • 500-4000 character output                        │
+│ • Syntax highlighting for steps/commands             │
+└─────────────────────────────────────────────────────┘
+```
+
+### Source Priority
+
+1. **DuckDuckGo** (Primary) - Real-time web results
+2. **Wikipedia** (Secondary) - Background concepts only
+3. **Web Scraping** - Full content from top 3
+
+## Export Location
+
+Summaries are saved to:
+- **Linux/macOS**: `~/Documents/ACHEM/`
+- **Windows**: `C:\Users\<username>\Documents\ACHEM\`
+
+## Disclaimer
+
+**ACHEM is for educational and research purposes only.**
+
+The tool aggregates publicly available information from the web. Any actions taken based on the information provided are the sole responsibility of the user. The developer is not responsible for any misuse of this tool.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit issues and pull requests.
+
+## Acknowledgments
+
+- [Hugging Face](https://huggingface.co/) - Free inference API
+- [DuckDuckGo](https://duckduckgo.com/) - Privacy-focused search
+- [Wikipedia](https://www.wikipedia.org/) - Free encyclopedia
+- [Qwen](https://huggingface.co/Qwen) - Open source AI models

achem-1.0.0/README.md

@@ -0,0 +1,208 @@
+# ACHEM - Deep Web Research Tool
+
+![ACHEM Banner](https://img.shields.io/badge/ACHEM-v1.0.0-blue?style=for-the-badge)
+
+> **ACHEM** (Arabic: آشم) is a powerful deep web research tool that aggregates information from 30+ sources, scrapes full content from top results, and generates concise summaries using AI.
+
+## Features
+
+- **Deep Web Research**: Gathers results from 30+ sources via DuckDuckGo
+- **Web Scraping**: Extracts full content from top 3 most relevant links
+- **Two-Pass Search**: Prioritizes technical content (StackOverflow, GitHub, forums)
+- **AI Summarization**: Uses Hugging Face Inference Providers (free tier)
+- **Syntax Highlighting**: Color-coded output for easy scanning
+- **SQLite Cache**: Instant recall for repeated searches
+- **Export**: Save summaries to Markdown files
+- **Multi-language**: Supports English, French, and Arabic
+
+## Screenshots
+
+```
+╔══════════════════════════════════════════════════════════════════╗
+║                    ACHEM - Deep Web Research                      ║
+╚══════════════════════════════════════════════════════════════════╝
+
+🔍 Deep Research: how to learn python
+==================================================
+PASS 1: Gathering 30 sources from DuckDuckGo...
+✓ Found 30 sources
+PASS 2: Scraped full content from top 3 links
+→ Analyzing 35 total sources...
+→ Generating deep summary...
+
+╭──────────────────────────────────────────────────────────────────╮
+│ UNIFIED RESEARCH SUMMARY                                          │
+├──────────────────────────────────────────────────────────────────┤
+│ 1. Start with the official Python tutorial:                     │
+│    - Visit docs.python.org/3/tutorial                           │
+│                                                                  │
+│ 2. Use free online tutorials:                                   │
+│    - LearnPython.org, pythonbasics.org                         │
+╰──────────────────────────────────────────────────────────────────╯
+```
+
+## Installation
+
+### Prerequisites
+
+- Python 3.10 or higher
+- pip package manager
+
+### Quick Install (PyPI)
+
+```bash
+pip install achem
+```
+
+### Or Install from Source
+
+1. **Clone the repository**
+```bash
+git clone https://github.com/achem/achem.git
+cd achem
+```
+
+2. **Install in editable mode**
+```bash
+pip install -e .
+```
+
+3. **Configure API keys**
+```bash
+cp .env.example .env
+```
+
+Then edit `.env` and add your Hugging Face API token:
+```env
+HF_API_KEY=hf_your_token_here
+HF_MODEL=Qwen/Qwen2.5-7B-Instruct
+```
+
+### Getting a Hugging Face API Token
+
+1. Go to [Hugging Face](https://huggingface.co/)
+2. Create an account (free)
+3. Go to Settings → Access Tokens
+4. Create a new token with "Read" permissions
+5. Copy the token to your `.env` file
+
+## Usage
+
+### Interactive Mode
+
+```bash
+python src/main.py
+```
+
+### Command Line Mode
+
+```bash
+python src/main.py "your search query"
+```
+
+### Options
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `-l, --limit` | Wikipedia results per query | 10 |
+| `--lang` | Language (en/fr/ar/auto) | auto |
+| `--ddg-limit` | DuckDuckGo results | 30 |
+| `--min-relevance` | Minimum relevance % | 0 |
+| `--no-cache` | Skip cache | False |
+| `--no-wikipedia` | Skip Wikipedia | False |
+| `--clear-cache` | Clear SQLite cache | False |
+
+### Commands (Interactive Mode)
+
+| Command | Description |
+|---------|-------------|
+| `clear` / `cls` | Clear screen |
+| `export` / `save` | Export last summary |
+| `help` / `?` | Show help |
+| `version` / `v` | Show version |
+| `exit` / `quit` / `q` | Exit program |
+
+## Project Structure
+
+```
+ACHEM/
+├── src/
+│   └── achem/              # Main package
+│       ├── __init__.py
+│       ├── main.py        # Entry point
+│       ├── commands.py    # Command handler
+│       ├── config_manager.py    # Config loader
+│       ├── duckduckgo_client.py # DDG search
+│       ├── export_manager.py    # Export to Documents/ACHEM/
+│       ├── huggingface_summarizer.py # AI summarization
+│       ├── output_formatter.py # Terminal UI
+│       ├── search_router.py    # Source priority
+│       ├── sqlite_cache.py     # SQLite cache
+│       ├── spell_checker.py    # Typo correction
+│       ├── text_analyzer.py    # TF-IDF analysis
+│       ├── user_input.py       # Input handler
+│       ├── web_scraper.py      # BeautifulSoup scraper
+│       └── wikipedia_client.py  # Wikipedia API
+├── .env.example            # Config template
+├── .gitignore
+├── LICENSE
+├── README.md
+└── pyproject.toml         # Package metadata
+```
+
+## How It Works
+
+### Two-Pass Search System
+
+```
+┌─────────────────────────────────────────────────────┐
+│ PASS 1: DuckDuckGo Search (30 results)              │
+│ • Prioritizes technical sites                        │
+│ • Filters out cookie/login/consent pages            │
+│ • Ranks by domain authority                         │
+├─────────────────────────────────────────────────────┤
+│ PASS 2: Web Scraping (Top 3)                        │
+│ • BeautifulSoup extracts full article text           │
+│ • Removes navigation/footer/scripts                 │
+│ • Combines up to 10,000 chars per article           │
+├─────────────────────────────────────────────────────┤
+│ PASS 3: AI Summarization                            │
+│ • Neutral technical prompt                           │
+│ • No ethical warnings or opinions                   │
+│ • 500-4000 character output                        │
+│ • Syntax highlighting for steps/commands             │
+└─────────────────────────────────────────────────────┘
+```
+
+### Source Priority
+
+1. **DuckDuckGo** (Primary) - Real-time web results
+2. **Wikipedia** (Secondary) - Background concepts only
+3. **Web Scraping** - Full content from top 3
+
+## Export Location
+
+Summaries are saved to:
+- **Linux/macOS**: `~/Documents/ACHEM/`
+- **Windows**: `C:\Users\<username>\Documents\ACHEM\`
+
+## Disclaimer
+
+**ACHEM is for educational and research purposes only.**
+
+The tool aggregates publicly available information from the web. Any actions taken based on the information provided are the sole responsibility of the user. The developer is not responsible for any misuse of this tool.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit issues and pull requests.
+
+## Acknowledgments
+
+- [Hugging Face](https://huggingface.co/) - Free inference API
+- [DuckDuckGo](https://duckduckgo.com/) - Privacy-focused search
+- [Wikipedia](https://www.wikipedia.org/) - Free encyclopedia
+- [Qwen](https://huggingface.co/Qwen) - Open source AI models

achem-1.0.0/pyproject.toml

@@ -0,0 +1,75 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "achem"
+version = "1.0.0"
+description = "Deep Web Research Tool - Aggregates 30+ sources, scrapes content, generates AI summaries"
+readme = "README.md"
+license = "MIT"
+authors = [
+    {name = "ACHEM Contributors"}
+]
+keywords = [
+    "research",
+    "deep-web",
+    "web-scraping",
+    "summarization",
+    "ai",
+    "cli",
+    "tool"
+]
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Education",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+    "Topic :: Text Processing :: General",
+]
+requires-python = ">=3.10"
+dependencies = [
+    "wikipedia-api>=0.5.4",
+    "rich>=13.0.0",
+    "psutil>=5.9.0",
+    "prompt_toolkit>=3.0.0",
+    "pyfiglet>=0.8.0",
+    "openai>=1.0.0",
+    "ddgs>=3.0.0",
+    "beautifulsoup4>=4.12.0",
+    "requests>=2.31.0",
+]
+
+[project.optional-dependencies]
+arabic = [
+    "arabic-reshaper>=3.0.0",
+    "python-bidi>=0.14.0",
+]
+dev = [
+    "pytest>=7.0.0",
+    "ruff>=0.1.0",
+]
+
+[project.scripts]
+achem = "achem.main:main"
+
+[project.urls]
+Homepage = "https://github.com/achem/achem"
+Documentation = "https://github.com/achem/achem#readme"
+Repository = "https://github.com/achem/achem"
+Issues = "https://github.com/achem/achem/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+achem = ["py.typed"]

achem-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

achem-1.0.0/src/achem/__init__.py

@@ -0,0 +1,9 @@
+"""ACHEM - Deep Web Research Tool
+
+A powerful CLI tool for aggregating information from 30+ sources,
+scraping full content, and generating AI-powered summaries.
+"""
+
+__version__ = "1.0.0"
+__author__ = "ACHEM Contributors"
+__license__ = "MIT"

achem-1.0.0/src/achem/__main__.py

@@ -0,0 +1,6 @@
+"""ACHEM - Run as: python -m achem"""
+
+from .main import main
+
+if __name__ == "__main__":
+    main()

achem-1.0.0/src/achem/cache_manager.py

@@ -0,0 +1,98 @@
+import os
+import json
+import time
+import hashlib
+from pathlib import Path
+from typing import Optional
+
+
+class CacheManager:
+    def __init__(self, cache_dir: str = None, ttl_seconds: int = 86400):
+        if cache_dir is None:
+            cache_dir = os.path.join(
+                os.path.expanduser("~"), ".wiki-summarizer", "cache"
+            )
+
+        self.cache_dir = Path(cache_dir)
+        self.cache_dir.mkdir(parents=True, exist_ok=True)
+        self.ttl_seconds = ttl_seconds
+
+    def _get_cache_key(self, query: str) -> str:
+        """Generate a unique cache key for a query."""
+        query_normalized = query.lower().strip()
+        return hashlib.md5(query_normalized.encode()).hexdigest()
+
+    def _get_cache_path(self, cache_key: str) -> Path:
+        """Get the file path for a cache key."""
+        return self.cache_dir / f"{cache_key}.json"
+
+    def get(self, query: str) -> Optional[dict]:
+        """Retrieve cached data for a query."""
+        cache_key = self._get_cache_key(query)
+        cache_path = self._get_cache_path(cache_key)
+
+        if not cache_path.exists():
+            return None
+
+        try:
+            with open(cache_path, "r", encoding="utf-8") as f:
+                cache_data = json.load(f)
+
+            cached_time = cache_data.get("timestamp", 0)
+            if time.time() - cached_time > self.ttl_seconds:
+                cache_path.unlink()
+                return None
+
+            return cache_data.get("data")
+        except (json.JSONDecodeError, IOError):
+            return None
+
+    def set(self, query: str, data: dict) -> None:
+        """Store data in cache for a query."""
+        cache_key = self._get_cache_key(query)
+        cache_path = self._get_cache_path(cache_key)
+
+        cache_data = {"query": query, "timestamp": time.time(), "data": data}
+
+        try:
+            with open(cache_path, "w", encoding="utf-8") as f:
+                json.dump(cache_data, f, ensure_ascii=False, indent=2)
+        except IOError:
+            pass
+
+    def invalidate(self, query: str = None) -> None:
+        """Invalidate cache for a specific query or all queries."""
+        if query:
+            cache_key = self._get_cache_key(query)
+            cache_path = self._get_cache_path(cache_key)
+            if cache_path.exists():
+                cache_path.unlink()
+        else:
+            for cache_file in self.cache_dir.glob("*.json"):
+                cache_file.unlink()
+
+    def get_stats(self) -> dict:
+        """Get cache statistics."""
+        total_size = 0
+        file_count = 0
+        expired_count = 0
+        current_time = time.time()
+
+        for cache_file in self.cache_dir.glob("*.json"):
+            file_count += 1
+            total_size += cache_file.stat().st_size
+
+            try:
+                with open(cache_file, "r", encoding="utf-8") as f:
+                    cache_data = json.load(f)
+                cached_time = cache_data.get("timestamp", 0)
+                if current_time - cached_time > self.ttl_seconds:
+                    expired_count += 1
+            except (json.JSONDecodeError, IOError):
+                pass
+
+        return {
+            "total_files": file_count,
+            "total_size_mb": round(total_size / (1024 * 1024), 2),
+            "expired_files": expired_count,
+        }