@avalw/search-worker 1.0.3 → 2.1.0

package/README.md

@@ -1,130 +1,303 @@
 # @avalw/search-worker
 
-Free privacy-focused search API in exchange for compute power.
-
-[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
-[![Node.js Version](https://img.shields.io/badge/node-%3E%3D14.0.0-brightgreen)](https://nodejs.org)
-
-## What is AVALW Worker?
+```
+    ___ _    _____ _   __      __
+   /   | |  / /   | | / /     / /
+  / /| | | / / /| | |/ / _ __/ /_
+ / ___ | |/ / ___ |   / | '__  __|
+/_/  |_|___/_/  |_|_|\_\|_|  |_|
 
-AVALW Worker is a lightweight Node.js application that runs on your server and contributes CPU power to the AVALW distributed network. In return, you get **free access** to the AVALW Search API - a privacy-focused search engine that doesn't track users. Your API quota scales with your contribution.
+ Distributed Cache Worker v2.1.0
+```
 
-**No tracking. No ads. No data collection. Just search.**
+Contribute your computer's cache storage to the AVALW search network and earn free API access in return.
 
 ## How It Works
 
 ```
-┌─────────────────┐      ┌─────────────────┐      ┌─────────────────┐
-│   Your Server   │      │  AVALW Network  │      │   Your Users    │
-│                 │      │                 │      │                 │
-│  ┌───────────┐  │      │                 │      │                 │
-│  │  Worker   │──┼─────►│  Contributes    │      │                 │
-│  │  Process  │  │      │  compute power  │      │                 │
-│  └───────────┘  │      │                 │      │                 │
-│                 │      │                 │      │                 │
-│  ┌───────────┐  │      │                 │      │  ┌───────────┐  │
-│  │  Your App │◄─┼──────┤  Search API     │◄─────┼──│  Search   │  │
-│  │           │  │      │  (free access)  │      │  │  Queries  │  │
-│  └───────────┘  │      │                 │      │  └───────────┘  │
-└─────────────────┘      └─────────────────┘      └─────────────────┘
+┌─────────────────────────────────────────────────────────────────────────┐
+│                        AVALW DISTRIBUTED CACHE                          │
+├─────────────────────────────────────────────────────────────────────────┤
+│                                                                         │
+│    ┌─────────┐         ┌──────────────┐         ┌─────────────┐        │
+│    │  User   │────────▶│  avalw.org   │────────▶│ Coordinator │        │
+│    │ Search  │         │   Server     │         │   Server    │        │
+│    └─────────┘         └──────────────┘         └──────┬──────┘        │
+│                                                        │               │
+│                              ┌─────────────────────────┼───────┐       │
+│                              │                         │       │       │
+│                              ▼                         ▼       ▼       │
+│                        ┌──────────┐            ┌──────────┐ ┌──────┐   │
+│                        │ Worker 1 │            │ Worker 2 │ │ ...  │   │
+│                        │  Cache   │            │  Cache   │ │      │   │
+│                        └──────────┘            └──────────┘ └──────┘   │
+│                                                                         │
+│    YOUR WORKER stores search results and serves them when requested     │
+│                                                                         │
+└─────────────────────────────────────────────────────────────────────────┘
 ```
 
-## Tiers
+### The Flow
 
-Choose the tier that fits your needs:
+```
+1. USER SEARCHES          2. CHECK CACHE           3. SERVE RESULT
+   "spotify"
+                          ┌─────────────┐
+   ┌──────┐              │ Coordinator  │         ┌──────────┐
+   │ User │──search──▶   │   checks     │──hit──▶ │  Worker  │──result──▶ User
+   └──────┘              │   workers    │         │  serves  │
+                         └─────────────┘          └──────────┘
+                               │
+                          miss │
+                               ▼
+                         ┌──────────┐
+                         │  Server  │──result──▶ User + Distribute to Workers
+                         │ process  │
+                         └──────────┘
+```
 
-| Tier       | Requests/Day | CPU Contribution |
-|------------|--------------|------------------|
-| Starter    | 1,000        | 2%               |
-| Basic      | 5,000        | 5%               |
-| Pro        | 20,000       | 10%              |
-| Business   | 50,000       | 20%              |
-| Enterprise | 500,000      | 50%              |
+### What Your Worker Does vs. Doesn't Do
+
+```
+   ✅ DOES                              ❌ DOESN'T
+   ─────────────────────────────────    ─────────────────────────────────
+   • Store search RESULTS               • Have the search INDEX
+   • Serve cached results               • Process raw documents
+   • Use 50-200MB RAM                   • Use significant CPU
+   • Use 500MB-2GB disk                 • Access your files
+   • Connect via WebSocket              • Run arbitrary code
+   • Earn you API access                • Collect your data
+```
 
 ## Quick Start
 
-### 1. Get Your Token
+```bash
+# 1. Get your token at https://worker.avalw.org
+# 2. Run the worker:
+npx @avalw/search-worker --token YOUR_TOKEN
+```
 
-Visit [worker.avalw.org](https://worker.avalw.org) and create a free account. You'll receive:
-- **Account Hash** - Your unique identifier (save this!)
-- **Worker Token** - Used to run the worker
+That's it! Your worker will connect and start contributing.
 
-### 2. Install the Worker
+## Installation
 
 ```bash
+# Global install
 npm install -g @avalw/search-worker
+
+# Or use npx (no install needed)
+npx @avalw/search-worker --token YOUR_TOKEN
 ```
 
-### 3. Run the Worker
+## What's New in v2.1.0
 
-```bash
-avalw-worker YOUR_WORKER_TOKEN
 ```
+┌────────────────────────────────────────────────────────────────┐
+│  INTELLIGENT CACHE SYSTEM                                      │
+├────────────────────────────────────────────────────────────────┤
+│                                                                │
+│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐            │
+│  │ Size Limits │  │ Time-Based  │  │ Popularity  │            │
+│  │ 100MB - 2GB │  │  Eviction   │  │  Scoring    │            │
+│  └─────────────┘  └─────────────┘  └─────────────┘            │
+│                                                                │
+│  • Cache up to 2GB of search results                          │
+│  • Auto-remove queries unused for 7+ days                     │
+│  • Popular queries stay cached longer                         │
+│  • Background cleanup every hour                              │
+│                                                                │
+└────────────────────────────────────────────────────────────────┘
+```
+
+## Command Line Options
+
+```bash
+# Basic usage (500MB cache, 7 day retention)
+npx @avalw/search-worker --token abc123
 
-You should see:
+# Custom cache size (100-2048 MB)
+npx @avalw/search-worker --token abc123 --cache-size 1024
+
+# Custom retention (1-30 days)
+npx @avalw/search-worker --token abc123 --max-age 14
+
+# Full customization
+npx @avalw/search-worker \
+  --token abc123 \
+  --cache-size 1024 \
+  --max-age 14
 ```
-  ╔═══════════════════════════════════════════════╗
-  ║         AVALW Search Worker v1.0.0            ║
-  ╠═══════════════════════════════════════════════╣
-  ║  Free Search API in exchange for compute      ║
-  ╚═══════════════════════════════════════════════╝
 
-  API URL: https://worker.avalw.org
-  Token:   abc123def456...
+### All Options
 
-  [10:30:45] Heartbeat OK | Tier: Starter | CPU Limit: 2%
-  Worker started. Press Ctrl+C to stop.
+```
+┌──────────────────┬─────────────────┬───────────────────────────────────┐
+│ Option           │ Default         │ Description                       │
+├──────────────────┼─────────────────┼───────────────────────────────────┤
+│ -t, --token      │ $AVALW_TOKEN    │ Your worker authentication token  │
+│ -s, --cache-size │ 500             │ Max cache size in MB (100-2048)   │
+│ -a, --max-age    │ 7               │ Days to keep unused queries (1-30)│
+│ -h, --help       │                 │ Show help                         │
+│ -v, --version    │                 │ Show version                      │
+└──────────────────┴─────────────────┴───────────────────────────────────┘
 ```
 
-### 4. Use the Search API
+### Environment Variables
 
-Once your worker is running, make API requests from your whitelisted domains:
+```bash
+export AVALW_WORKER_TOKEN=your-token-here
+export AVALW_CACHE_SIZE_MB=1024
+export AVALW_MAX_AGE_DAYS=14
+npx @avalw/search-worker
+```
+
+## Programmatic Usage
 
 ```javascript
-// Node.js / JavaScript
-const response = await fetch(
-  'https://worker.avalw.org/api/search?q=your+query&token=YOUR_TOKEN'
-);
-const results = await response.json();
-console.log(results);
+const { createWorker } = require('@avalw/search-worker');
+
+const worker = createWorker({
+  workerToken: 'your-token-here',
+
+  // Cache configuration
+  cacheSizeMB: 1024,       // 1GB cache
+  maxAgeDays: 14,          // 14 day retention
+
+  // Event handlers
+  onConnected: (w) => console.log('Connected to coordinator'),
+  onDisconnected: (w) => console.log('Disconnected'),
+  onCacheHit: (key) => console.log('Served:', key),
+  onCacheStore: (key) => console.log('Cached:', key),
+  onError: (err) => console.error('Error:', err)
+});
+
+// Connect
+worker.connect();
+
+// Get stats
+const stats = worker.getStats();
+console.log(`Cache: ${stats.cache.sizeFormatted} / ${stats.cache.maxSizeFormatted}`);
+console.log(`Hit rate: ${stats.hitRate}%`);
+
+// Runtime configuration
+worker.setCacheSize(2048);  // Change to 2GB
+worker.setCacheMaxAge(30);  // Change to 30 days
+
+// Cleanup
+worker.disconnect();
 ```
 
-```python
-# Python
-import requests
+## API Tiers & Rewards
 
-response = requests.get(
-    'https://worker.avalw.org/api/search',
-    params={'q': 'your query', 'token': 'YOUR_TOKEN'}
-)
-results = response.json()
-print(results)
+Your contribution is measured in Core-Hours:
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                         CORE-HOURS FORMULA                          │
+├─────────────────────────────────────────────────────────────────────┤
+│                                                                     │
+│   Core-Hours = Cores × (CPU Speed ÷ 3.0 GHz) × Usage% × Hours      │
+│                                                                     │
+│   Example: 8 cores × (3.5 GHz ÷ 3.0 GHz) × 80% × 1 hour            │
+│          = 8 × 1.17 × 0.8 × 1 = 7.5 Core-Hours                     │
+│                                                                     │
+└─────────────────────────────────────────────────────────────────────┘
 ```
 
-```bash
-# cURL
-curl "https://worker.avalw.org/api/search?q=your+query&token=YOUR_TOKEN"
+### Tier Benefits
+
+```
+┌─────────────┬────────────┬───────────┬─────────────────┐
+│ Tier        │ Core-Hours │ Min Cores │ API Requests    │
+├─────────────┼────────────┼───────────┼─────────────────┤
+│ Starter     │ 3          │ 3         │ 1,000/day       │
+│ Basic       │ 12         │ 4         │ 5,000/day       │
+│ Pro         │ 36         │ 6         │ 20,000/day      │
+│ Business    │ 72         │ 8         │ 50,000/day      │
+│ Enterprise  │ 240        │ 16        │ 500,000/day     │
+└─────────────┴────────────┴───────────┴─────────────────┘
 ```
 
-## Running as a Service
+## Intelligent Cache Details
 
-### Using PM2 (Recommended)
+### How Eviction Works
 
-```bash
-# Install PM2
-npm install -g pm2
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    EVICTION PRIORITY                        │
+├─────────────────────────────────────────────────────────────┤
+│                                                             │
+│  1. EXPIRED TTL                                             │
+│     └── Items past their time-to-live (5 min default)      │
+│                                                             │
+│  2. MAX AGE EXCEEDED                                        │
+│     └── Items not accessed for N days (7 days default)     │
+│                                                             │
+│  3. LOWEST POPULARITY                                       │
+│     └── Score = log2(accessCount) × recency × stability    │
+│                                                             │
+│  Priority: 1 > 2 > 3 (expired items removed first)         │
+│                                                             │
+└─────────────────────────────────────────────────────────────┘
+```
 
-# Start worker
-pm2 start avalw-worker -- YOUR_WORKER_TOKEN
+### Popularity Scoring
 
-# Save for auto-restart
-pm2 save
-pm2 startup
+```
+Score = log2(accessCount + 1) × recencyFactor × stabilityFactor
+
+┌─────────────────┬─────────────────────────────────────────┐
+│ Factor          │ Description                             │
+├─────────────────┼─────────────────────────────────────────┤
+│ accessCount     │ Number of times the query was accessed  │
+│ recencyFactor   │ 1.0 → 0.0 over 7 days since last access │
+│ stabilityFactor │ Higher for consistent access patterns   │
+└─────────────────┴─────────────────────────────────────────┘
+
+Example: Query accessed 100 times, last accessed 1 day ago
+Score = log2(101) × 0.86 × 1.2 = 6.65 × 0.86 × 1.2 = 6.86
 ```
 
-### Using systemd (Linux)
+## Resource Usage
 
-Create `/etc/systemd/system/avalw-worker.service`:
+```
+┌──────────────────────────────────────────────────────────┐
+│                 TYPICAL RESOURCE USAGE                   │
+├──────────────────────────────────────────────────────────┤
+│                                                          │
+│  CPU:     1-5%     [▓░░░░░░░░░░░░░░░░░░░] minimal       │
+│  RAM:     50-200MB [▓▓▓▓░░░░░░░░░░░░░░░░] light         │
+│  Disk:    500MB-2GB [▓▓▓▓▓▓▓▓░░░░░░░░░░░░] configurable │
+│  Network: Low       [▓▓░░░░░░░░░░░░░░░░░░] WebSocket    │
+│                                                          │
+└──────────────────────────────────────────────────────────┘
+```
+
+## Privacy & Security
+
+```
+┌────────────────────────────────────────────────────────────────┐
+│                      PRIVACY GUARANTEES                        │
+├────────────────────────────────────────────────────────────────┤
+│                                                                │
+│  ✓ Workers cache RESULTS only, not the search index           │
+│  ✓ No IP addresses are logged                                 │
+│  ✓ Workers don't see who requested a search                   │
+│  ✓ All communication encrypted (WSS/HTTPS)                    │
+│  ✓ No access to your local files                              │
+│  ✓ Open source - audit the code yourself                      │
+│                                                                │
+└────────────────────────────────────────────────────────────────┘
+```
+
+## Running as a Service
+
+### Linux (systemd)
+
+```bash
+# Create service file
+sudo nano /etc/systemd/system/avalw-worker.service
+```
 
 ```ini
 [Unit]
@@ -133,8 +306,11 @@ After=network.target
 
 [Service]
 Type=simple
-User=your-user
-ExecStart=/usr/bin/avalw-worker YOUR_WORKER_TOKEN
+User=nobody
+Environment=AVALW_WORKER_TOKEN=your-token-here
+Environment=AVALW_CACHE_SIZE_MB=1024
+Environment=AVALW_MAX_AGE_DAYS=14
+ExecStart=/usr/bin/npx @avalw/search-worker
 Restart=always
 RestartSec=10
 
@@ -142,140 +318,103 @@ RestartSec=10
 WantedBy=multi-user.target
 ```
 
-Then:
 ```bash
+# Enable and start
+sudo systemctl daemon-reload
 sudo systemctl enable avalw-worker
 sudo systemctl start avalw-worker
+
+# Check status
+sudo systemctl status avalw-worker
 ```
 
-### Using Docker
+### Docker
 
 ```dockerfile
-FROM node:20-alpine
+FROM node:18-alpine
 RUN npm install -g @avalw/search-worker
-CMD ["avalw-worker", "YOUR_WORKER_TOKEN"]
+ENV AVALW_WORKER_TOKEN=your-token-here
+ENV AVALW_CACHE_SIZE_MB=1024
+ENV AVALW_MAX_AGE_DAYS=14
+CMD ["avalw-worker"]
 ```
 
 ```bash
 docker build -t avalw-worker .
-docker run -d --name avalw-worker avalw-worker
+docker run -d --name avalw avalw-worker
 ```
 
-## Programmatic Usage
-
-You can also use the worker programmatically in your Node.js application:
-
-```javascript
-const { AvalwWorker } = require('@avalw/search-worker');
-
-const worker = new AvalwWorker('YOUR_WORKER_TOKEN');
+### PM2
 
-// Start the worker
-await worker.start();
+```bash
+pm2 start avalw-worker --name "avalw" -- --token YOUR_TOKEN --cache-size 1024
+pm2 save
+pm2 startup
+```
 
-// Get system info
-const info = worker.getSystemInfo();
-console.log(info);
+### Windows (Task Scheduler)
 
-// Stop the worker
-await worker.stop();
+```batch
+@echo off
+set AVALW_WORKER_TOKEN=your-token-here
+set AVALW_CACHE_SIZE_MB=1024
+npx @avalw/search-worker
 ```
 
-## API Reference
-
-### Search Endpoint
+## Troubleshooting
 
 ```
-GET https://worker.avalw.org/api/search
+┌─────────────────────────────────────────────────────────────────┐
+│                      COMMON ISSUES                              │
+├─────────────────────────────────────────────────────────────────┤
+│                                                                 │
+│  "Connection refused"                                           │
+│  └── Check internet connection, firewall settings               │
+│                                                                 │
+│  "Invalid token"                                                │
+│  └── Get a new token at https://worker.avalw.org                │
+│                                                                 │
+│  "Cache full, high eviction rate"                               │
+│  └── Increase --cache-size or decrease --max-age                │
+│                                                                 │
+│  "Memory usage too high"                                        │
+│  └── Decrease --cache-size (minimum 100MB)                      │
+│                                                                 │
+│  "Worker keeps disconnecting"                                   │
+│  └── Check network stability, worker will auto-reconnect        │
+│                                                                 │
+└─────────────────────────────────────────────────────────────────┘
 ```
 
-**Parameters:**
-| Parameter | Type   | Required | Description                    |
-|-----------|--------|----------|--------------------------------|
-| q         | string | Yes      | Search query                   |
-| token     | string | Yes      | Your worker token              |
-| page      | number | No       | Page number (default: 1)       |
-| limit     | number | No       | Results per page (default: 10) |
+## System Requirements
 
-**Response:**
-```json
-{
-  "success": true,
-  "results": [
-    {
-      "title": "Example Result",
-      "url": "https://example.com",
-      "snippet": "This is a snippet..."
-    }
-  ],
-  "total": 100,
-  "page": 1
-}
 ```
-
-## Domain Whitelisting
-
-For security, API requests only work from whitelisted domains. Manage your domains at [worker.avalw.org](https://worker.avalw.org) dashboard.
-
-## Troubleshooting
-
-### Worker won't connect
-- Check your internet connection
-- Verify your token is correct
-- Ensure port 443 (HTTPS) is not blocked
-
-### API returns "Worker offline"
-- Make sure your worker is running
-- Check the worker console for errors
-- Worker must send heartbeat every 30 seconds
-
-### Rate limit exceeded
-- Upgrade to a higher tier for more requests
-- Wait until the daily limit resets (midnight UTC)
-
-## Security
-
-- **No data collection**: We don't log search queries
-- **No tracking**: No cookies, no fingerprinting
-- **Open protocol**: Transparent heartbeat mechanism
-- **Your server, your control**: Worker runs on your infrastructure
+┌────────────────┬────────────────────────────────────┐
+│ Component      │ Requirement                        │
+├────────────────┼────────────────────────────────────┤
+│ Node.js        │ 14.0.0 or higher                   │
+│ CPU            │ 3+ cores for Starter tier          │
+│ RAM            │ 512MB+ free                        │
+│ Disk           │ 500MB - 2GB for cache              │
+│ Network        │ Stable internet, WebSocket support │
+│ OS             │ Windows, macOS, Linux              │
+└────────────────┴────────────────────────────────────┘
+```
 
 ## Support
 
-- Website: [worker.avalw.org](https://worker.avalw.org)
+- Dashboard: [worker.avalw.org](https://worker.avalw.org)
+- Documentation: [worker.avalw.org/protocol](https://worker.avalw.org/protocol)
 - Email: contact@avalw.com
+- Search Engine: [avalw.org](https://avalw.org)
 
 ## License
 
-MIT License
-
-Copyright (c) 2026 AVALW
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License - Free to use, modify, and distribute.
 
 ---
 
-Made with privacy in mind by [AVALW](https://cronos.avalw.com/about)
-
-### Our Projects
-
-- [AVALW Search](https://avalw.org) - Privacy-focused search engine
-- [Cronos Browser](https://cronos.avalw.com) - Secure, private browser
-- [AVALW Protocol](https://avalw.com) - Decentralized communication protocol
-- [Worker API](https://worker.avalw.org) - This project
+```
+Made with transparency in mind by AVALW
+https://avalw.com
+```