prospector_engine 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 46798902589ee83ac3a63b7cf36fcb67b61d4a84fbf2ac07819b04a778439099
-  data.tar.gz: 87cb71c8d3c832156306184d31f9a7d3e8f45343e33bceb266ae3cc095a61130
+  metadata.gz: 8e754e44af2ba6797161e333fc790c36f3a5e8b79e30deb4dc5eef78ddb820c1
+  data.tar.gz: cf8338a50a7dfb29f2814b250dea974d69306c1e5c3ae7d69272663b4cd8cd43
 SHA512:
-  metadata.gz: 68982b0d84db8931d2e691f365d0032a5bfbff4f0f9ea40fcafcabeab8e0783699bb8c6e202d2529c083fa639e22ac77aae961978ddb7b0663d4414f9f255868
-  data.tar.gz: 1876729cef3a17d3a09a717de8ef392c4e67a611b6d164d36b0b985e37dda846f16d3814273dabbfe551a01adae5d9b6999f577b88ce82a389bc102ba57d46de
+  metadata.gz: dc08f57c755577e9589a729e671d21b3729dad5a81a6e0b8129ffe432eb07edaaaba8a0ad755827c45247616f2a11517a0750ed295d5bdb498c22461401b6648
+  data.tar.gz: 22026bf0d973173b0c22d4b3c4648a0ac9957b83ae8dd164dab0cee6e3256bb3aa29f64a101207ed3f8084d1416337713df738835d6e1416abba16d16dfe8113

data/README.md

@@ -1,20 +1,21 @@
 # Prospector
 
-A Rails engine for discovering businesses from multiple sources with AI-powered keyword generation and classification.
+A Rails engine for discovering businesses from multiple sources with AI-powered keyword generation, contact enrichment, and classification.
 
-Prospector handles the full discovery pipeline: generate search keywords with AI, fetch business listings from external APIs, classify results for domain relevance, and present an admin review interface -- all inside your existing Rails app.
+Prospector handles the full discovery pipeline: generate search keywords with AI, fetch business listings from external APIs, scrape websites for contact info, classify results for domain relevance, and present an admin review interface -- all inside your existing Rails app.
 
 ## Features
 
-- **Multi-source adapters** -- Ships with Google Places. Pluggable interface for adding Yelp, Bing, OpenStreetMap, or custom sources.
+- **Multi-source adapters** -- Ships with Google Places. Pluggable interface for adding Yelp, Bing, or custom sources.
+- **Contact enrichment** -- Automatically scrapes candidate websites for email and social links (Facebook, Instagram, LinkedIn, TikTok, YouTube). SSRF-protected with IP validation on every redirect hop.
 - **AI keyword generation** -- On-demand LLM-powered keyword generation for any business domain. Keywords are stored and reused across runs.
 - **AI classification** -- Automatically classifies discovered businesses for domain relevance. Non-relevant results are auto-rejected.
-- **Flexible geography** -- Search by metro area, city, coordinates + radius, ZIP code, or bounding box.
-- **Admin UI** -- Mountable admin interface with self-contained CSS. Review candidates, approve/reject, bulk approve, trigger reclassification.
+- **Flexible geography** -- Search by metro area, city, coordinates + radius, ZIP code, or bounding box. 42 preloaded US metro areas available as presets.
+- **Admin UI** -- Mountable admin interface with self-contained dark theme. Review candidates, approve/reject, bulk approve, trigger reclassification.
 - **Keyword management** -- Admin UI for viewing, adding, toggling, and AI-generating search keywords per category.
 - **Background jobs** -- Fetch, classify, and bulk approve jobs integrate with your existing queue (Solid Queue, Sidekiq, etc.).
-- **Turbo Streams** -- Real-time updates when Turbo is present. Gracefully degrades without it.
-- **Domain-agnostic** -- Works for motorcycle services, aviation operators, restaurants, or any business vertical.
+- **Turbo Streams** -- Real-time progress updates when Turbo is present. Gracefully degrades without it.
+- **Instrumentation** -- ActiveSupport::Notifications events for run status changes and candidate approvals.
 
 ## Requirements
 
@@ -51,11 +52,20 @@ Edit `config/initializers/prospector.rb`:
 
 ```ruby
 Prospector.configure do |config|
-  # Required: domain slug for keyword generation and classification context
+  # Required: domain slug for keyword generation and classification context.
   config.domain = "motorcycle_services"
 
-  # Required: called when an admin approves a candidate.
-  # Receives a Prospector::Candidate instance.
+  # Required: admin authentication.
+  config.authenticate_admin_with do |controller|
+    controller.current_user&.admin?
+  end
+
+  # Classifier class (must be set before classification runs).
+  # Can be a class constant or a string for lazy resolution.
+  config.classifier = MotorcycleClassifier
+  # config.classifier = "MotorcycleClassifier"
+
+  # Called when an admin approves a candidate (optional).
   # Runs AFTER the candidate status is committed (safe to enqueue jobs).
   config.on_approve do |candidate|
     data = candidate.normalized_data
@@ -67,26 +77,16 @@ Prospector.configure do |config|
       zip_code: data["zip_code"],
       phone_number: data["phone_number"],
       website: data["website"],
+      email: candidate.email,
       latitude: data["latitude"],
       longitude: data["longitude"],
       service_types: candidate.llm_categories,
-      import_source: "prospector"
+      facebook_url: candidate.facebook_url,
+      instagram_url: candidate.instagram_url
     )
   end
 
-  # Required: admin authentication.
-  # Receives the controller instance.
-  config.authenticate_admin_with do |controller|
-    controller.current_user&.admin?
-  end
-
-  # Required: classifier class (inherits LlmClassifier::Classifier).
-  # Defines categories, model, and classification rules for your domain.
-  # See "Defining a Classifier" below.
-  config.classifier = MotorcycleClassifier
-
   # Optional: check for duplicates before creating candidates.
-  # Return true to skip the candidate.
   config.duplicate_check do |source_uid:, name:, **|
     Business.exists?(["import_metadata->>'place_id' = ?", source_uid])
   end
@@ -94,7 +94,7 @@ Prospector.configure do |config|
   # Optional: default source adapter (default: :google_places)
   config.default_source = :google_places
 
-  # Optional: default AI model (passed to classifier when not overridden)
+  # Optional: default AI model for classification and keyword generation
   config.default_classifier_model = "anthropic:claude-sonnet-4-20250514"
 
   # Optional: job queue name (default: :default)
@@ -102,6 +102,8 @@ Prospector.configure do |config|
 end
 ```
 
+**Note:** `validate!` only runs in production. In development, missing `domain` or `authenticate_admin_with` will cause runtime errors rather than startup failures.
+
 Set the required environment variables:
 
 ```bash
@@ -116,20 +118,21 @@ ANTHROPIC_API_KEY=your_anthropic_api_key  # or whatever ruby_llm needs
 Visit `/prospector` in your browser. The admin interface provides:
 
 - **Runs** -- Create, monitor, retry, restart, and cancel discovery runs
-- **Candidates** -- Review discovered businesses, approve or reject individually or in bulk
+- **Candidates** -- Review discovered businesses with contact info, approve or reject individually or in bulk
 - **Keywords** -- View, add, toggle, and AI-generate search keywords per category
 
 ### Creating a Run
 
 1. Click "New Run" in the admin UI
 2. Select a geography type (metro area, city, coordinates, ZIP code, or bounding box)
-3. Fill in the geography details
-4. Click "Start Run"
+3. For metro areas, optionally select from 42 preloaded US metros or enter a custom one
+4. Fill in the geography details
+5. Click "Start Run"
 
 The run progresses through these stages automatically:
 
 ```
-pending -> running (fetching from source) -> classifying (AI classification) -> completed
+pending -> running (fetch + enrich) -> classifying (AI) -> completed
 ```
 
 ### Defining a Classifier
@@ -145,7 +148,7 @@ class MotorcycleClassifier < LlmClassifier::Classifier
 
   model "anthropic:claude-sonnet-4-20250514"
   multi_label true
-  require_categories true  # fail when no categories match (auto-reject)
+  require_categories true
 
   system_prompt <<~PROMPT
     You are classifying businesses for the motorcycle services domain.
@@ -164,13 +167,7 @@ class MotorcycleClassifier < LlmClassifier::Classifier
 end
 ```
 
-Then set it in your Prospector config:
-
-```ruby
-config.classifier = MotorcycleClassifier
-```
-
-The classifier receives a hash with `name`, `address`, `website`, `description`, and `source_types` keys. It returns an `LlmClassifier::Result` with `.categories`, `.confidence`, and `.reasoning`.
+The classifier receives a hash with `name`, `address`, `website`, `description`, and `source_types` (comma-joined string) keys.
 
 ### Programmatic Usage
 
@@ -195,13 +192,19 @@ Prospector::Keyword.create!(
 # Approve a candidate programmatically
 candidate = Prospector::Candidate.find(id)
 candidate.approve!  # fires the on_approve callback
+
+# Access enriched contact info
+candidate.email          # => "info@example.com"
+candidate.facebook_url   # => "https://facebook.com/example"
+candidate.instagram_url  # => "https://instagram.com/example"
 ```
 
 ### Geography Types
 
 ```ruby
-# Metro area (text search)
-Prospector::Geography::MetroArea.new(name: "San Francisco", primary_state: "CA")
+# Metro area (text search) -- 42 US metros available as presets
+Prospector::Geography::MetroArea.new(name: "San Francisco Bay Area", primary_state: "CA")
+Prospector::Geography::MetroArea::PRELOADED  # => [{name: "Atlanta", primary_state: "GA"}, ...]
 
 # City (text search)
 Prospector::Geography::City.new(city: "Austin", state: "TX")
@@ -226,8 +229,6 @@ class MyApp::YelpAdapter < Prospector::Sources::Base
   def self.adapter_key = "yelp"
 
   def fetch(geography:, keywords:)
-    # Call the Yelp API for each keyword + geography combination.
-    # Return an Array of Prospector::Sources::Result.
     keywords.flat_map do |keyword|
       search_yelp(keyword, geography).map do |biz|
         Prospector::Sources::Result.new(
@@ -260,8 +261,6 @@ Prospector.configure do |config|
 end
 ```
 
-Then select "Yelp" as the source when creating a run.
-
 ## Database Tables
 
 Prospector creates four tables, all prefixed with `prospector_`:
@@ -269,7 +268,7 @@ Prospector creates four tables, all prefixed with `prospector_`:
 | Table | Purpose |
 |-------|---------|
 | `prospector_runs` | Discovery runs with status, geography, and progress counters |
-| `prospector_candidates` | Discovered businesses pending review |
+| `prospector_candidates` | Discovered businesses with contact info (email, social links) pending review |
 | `prospector_classification_runs` | Tracks AI reclassification operations |
 | `prospector_keywords` | Stored search keywords per domain and category |
 
@@ -284,33 +283,51 @@ Prospector::Run (state machine)
   |     |-- Pipeline::Orchestrator
   |     |     |-- Pipeline::Normalizer (address parsing)
   |     |     |-- Duplicate checking (via config.duplicate_check)
-  |     |     '-- Creates Prospector::Candidate records
+  |     |     |-- Creates Prospector::Candidate records
+  |     |     '-- Enrichment::ContactScraper (email + social links)
   |     '-- Enqueues ClassifyJob
   |
   |-- ClassifyJob
   |     '-- Classification::Runner
-  |           |-- Calls LLM via ruby_llm
+  |           |-- Calls LLM via llm_classifier
   |           |-- Stores categories, confidence, reasoning in metadata
   |           '-- Auto-rejects candidates with no relevant categories
   |
   '-- Admin UI
-        |-- Review candidates (approve / reject / restore)
-        |-- Bulk approve
-        '-- Trigger reclassification with different models
+        |-- Runs: create, monitor, retry, restart, cancel
+        |-- Candidates: review, approve, reject, restore, bulk approve
+        |-- Keywords: view, add, toggle active, AI-generate
+        '-- Reclassification with model override
 ```
 
 ## Run States
 
 ```
 pending --> running --> classifying --> completed
-               |                          |
-               v                          v
-            failed                    cancelled
-               ^                          ^
-               |__________________________|
-                    (retry / restart)
+    |          |            |
+    v          v            v
+             failed      cancelled
+               ^            ^
+               |____________|
+              (retry/restart)
 ```
 
+- `cancellable?` -- pending, running, or classifying
+- `retryable?` -- failed only
+- `restartable?` -- completed, failed, or cancelled
+- `restart!` destroys pending/rejected candidates and resets all counters
+- `reset_for_retry!` resets status and error fields only
+
+## Instrumentation
+
+Prospector emits ActiveSupport::Notifications events:
+
+| Event | Payload |
+|-------|---------|
+| `prospector.run.status_changed` | `run`, `status`, `previous_status` |
+| `prospector.candidate.approved` | `candidate`, `record` |
+| `prospector.candidate.rejected` | `candidate`, `reason` |
+
 ## Development
 
 The gem ships with a devcontainer for containerized development:
@@ -320,10 +337,10 @@ The gem ships with a devcontainer for containerized development:
 docker compose -f .devcontainer/compose.yaml up -d --build
 
 # Run tests (inside the container)
-docker exec -w /workspaces/prospector prospector-app-1 bundle exec rake test
+docker exec -w /workspaces/prospector_engine prospector_engine-app-1 bundle exec rake test
 
 # Run a single test file
-docker exec -w /workspaces/prospector prospector-app-1 bundle exec ruby -Itest test/models/run_test.rb
+docker exec -w /workspaces/prospector_engine prospector_engine-app-1 bundle exec ruby -Itest test/models/run_test.rb
 ```
 
 The test suite uses a dummy Rails app (`test/dummy/`) with PostgreSQL.

data/lib/prospector/engine.rb

@@ -2,8 +2,16 @@ module Prospector
   class Engine < ::Rails::Engine
     isolate_namespace Prospector
 
-    config.autoload_paths << File.expand_path("..", __dir__)
-    config.eager_load_paths << File.expand_path("..", __dir__)
+    initializer "prospector.autoload", before: :set_autoload_paths do |app|
+      lib_path = root.join("lib")
+      app.config.autoload_paths << lib_path
+      app.config.eager_load_paths << lib_path
+
+      Rails.autoloaders.main.ignore(
+        root.join("lib/prospector/version.rb"),
+        root.join("lib/generators")
+      )
+    end
 
     initializer "prospector.configuration" do
       Prospector.config.validate! if Rails.env.production?

data/lib/prospector/version.rb

@@ -1,3 +1,3 @@
 module Prospector
-  VERSION = "0.1.1"
+  VERSION = "0.1.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: prospector_engine
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
 platform: ruby
 authors:
 - AxiumFoundry