prospector_engine 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 46798902589ee83ac3a63b7cf36fcb67b61d4a84fbf2ac07819b04a778439099
+  data.tar.gz: 87cb71c8d3c832156306184d31f9a7d3e8f45343e33bceb266ae3cc095a61130
+SHA512:
+  metadata.gz: 68982b0d84db8931d2e691f365d0032a5bfbff4f0f9ea40fcafcabeab8e0783699bb8c6e202d2529c083fa639e22ac77aae961978ddb7b0663d4414f9f255868
+  data.tar.gz: 1876729cef3a17d3a09a717de8ef392c4e67a611b6d164d36b0b985e37dda846f16d3814273dabbfe551a01adae5d9b6999f577b88ce82a389bc102ba57d46de

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2026 AxiumFoundry
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,333 @@
+# Prospector
+
+A Rails engine for discovering businesses from multiple sources with AI-powered keyword generation and classification.
+
+Prospector handles the full discovery pipeline: generate search keywords with AI, fetch business listings from external APIs, classify results for domain relevance, and present an admin review interface -- all inside your existing Rails app.
+
+## Features
+
+- **Multi-source adapters** -- Ships with Google Places. Pluggable interface for adding Yelp, Bing, OpenStreetMap, or custom sources.
+- **AI keyword generation** -- On-demand LLM-powered keyword generation for any business domain. Keywords are stored and reused across runs.
+- **AI classification** -- Automatically classifies discovered businesses for domain relevance. Non-relevant results are auto-rejected.
+- **Flexible geography** -- Search by metro area, city, coordinates + radius, ZIP code, or bounding box.
+- **Admin UI** -- Mountable admin interface with self-contained CSS. Review candidates, approve/reject, bulk approve, trigger reclassification.
+- **Keyword management** -- Admin UI for viewing, adding, toggling, and AI-generating search keywords per category.
+- **Background jobs** -- Fetch, classify, and bulk approve jobs integrate with your existing queue (Solid Queue, Sidekiq, etc.).
+- **Turbo Streams** -- Real-time updates when Turbo is present. Gracefully degrades without it.
+- **Domain-agnostic** -- Works for motorcycle services, aviation operators, restaurants, or any business vertical.
+
+## Requirements
+
+- Ruby >= 3.1
+- Rails >= 7.1
+- PostgreSQL (uses `jsonb` columns)
+- A Google Places API key (for the built-in adapter)
+- An LLM API key compatible with [ruby_llm](https://github.com/crmne/ruby_llm) (for keyword generation and classification)
+- A classifier class inheriting from [LlmClassifier::Classifier](https://github.com/AxiumFoundry/llm_classifier)
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "prospector_engine", github: "AxiumFoundry/prospector_engine"
+```
+
+Run the install generator:
+
+```bash
+bin/rails generate prospector:install
+bin/rails db:migrate
+```
+
+This creates:
+- `config/initializers/prospector.rb` -- configuration file
+- A migration for the four Prospector tables
+- A route mounting the engine at `/prospector`
+
+## Configuration
+
+Edit `config/initializers/prospector.rb`:
+
+```ruby
+Prospector.configure do |config|
+  # Required: domain slug for keyword generation and classification context
+  config.domain = "motorcycle_services"
+
+  # Required: called when an admin approves a candidate.
+  # Receives a Prospector::Candidate instance.
+  # Runs AFTER the candidate status is committed (safe to enqueue jobs).
+  config.on_approve do |candidate|
+    data = candidate.normalized_data
+    Business.create!(
+      business_name: data["business_name"],
+      street_address: data["street_address"],
+      city: data["city"],
+      state: data["state"],
+      zip_code: data["zip_code"],
+      phone_number: data["phone_number"],
+      website: data["website"],
+      latitude: data["latitude"],
+      longitude: data["longitude"],
+      service_types: candidate.llm_categories,
+      import_source: "prospector"
+    )
+  end
+
+  # Required: admin authentication.
+  # Receives the controller instance.
+  config.authenticate_admin_with do |controller|
+    controller.current_user&.admin?
+  end
+
+  # Required: classifier class (inherits LlmClassifier::Classifier).
+  # Defines categories, model, and classification rules for your domain.
+  # See "Defining a Classifier" below.
+  config.classifier = MotorcycleClassifier
+
+  # Optional: check for duplicates before creating candidates.
+  # Return true to skip the candidate.
+  config.duplicate_check do |source_uid:, name:, **|
+    Business.exists?(["import_metadata->>'place_id' = ?", source_uid])
+  end
+
+  # Optional: default source adapter (default: :google_places)
+  config.default_source = :google_places
+
+  # Optional: default AI model (passed to classifier when not overridden)
+  config.default_classifier_model = "anthropic:claude-sonnet-4-20250514"
+
+  # Optional: job queue name (default: :default)
+  config.queue_name = :prospector
+end
+```
+
+Set the required environment variables:
+
+```bash
+GOOGLE_MAPS_API_KEY=your_google_places_api_key
+ANTHROPIC_API_KEY=your_anthropic_api_key  # or whatever ruby_llm needs
+```
+
+## Usage
+
+### Admin UI
+
+Visit `/prospector` in your browser. The admin interface provides:
+
+- **Runs** -- Create, monitor, retry, restart, and cancel discovery runs
+- **Candidates** -- Review discovered businesses, approve or reject individually or in bulk
+- **Keywords** -- View, add, toggle, and AI-generate search keywords per category
+
+### Creating a Run
+
+1. Click "New Run" in the admin UI
+2. Select a geography type (metro area, city, coordinates, ZIP code, or bounding box)
+3. Fill in the geography details
+4. Click "Start Run"
+
+The run progresses through these stages automatically:
+
+```
+pending -> running (fetching from source) -> classifying (AI classification) -> completed
+```
+
+### Defining a Classifier
+
+Prospector uses [llm_classifier](https://github.com/AxiumFoundry/llm_classifier) for AI classification. Define a classifier for your domain:
+
+```ruby
+# app/classifiers/motorcycle_classifier.rb
+class MotorcycleClassifier < LlmClassifier::Classifier
+  categories :mechanic, :instructor, :gear, :dealership, :storage,
+             :detailing, :training, :towing, :insurance, :touring,
+             :clubs, :parts
+
+  model "anthropic:claude-sonnet-4-20250514"
+  multi_label true
+  require_categories true  # fail when no categories match (auto-reject)
+
+  system_prompt <<~PROMPT
+    You are classifying businesses for the motorcycle services domain.
+    Determine which categories this business belongs to based on its
+    name, address, website, and any other available information.
+  PROMPT
+
+  knowledge do
+    motorcycle_brands %w[Harley-Davidson Honda Yamaha Kawasaki Suzuki Ducati Triumph]
+    service_descriptions({
+      mechanic: "Motorcycle repair and maintenance",
+      instructor: "Motorcycle riding lessons and schools",
+      gear: "Motorcycle apparel and equipment retail"
+    })
+  end
+end
+```
+
+Then set it in your Prospector config:
+
+```ruby
+config.classifier = MotorcycleClassifier
+```
+
+The classifier receives a hash with `name`, `address`, `website`, `description`, and `source_types` keys. It returns an `LlmClassifier::Result` with `.categories`, `.confidence`, and `.reasoning`.
+
+### Programmatic Usage
+
+```ruby
+# Create and start a run
+run = Prospector::Run.create!(
+  source_adapter: "google_places",
+  geography_type: "metro_area",
+  geography_data: { "name" => "Dallas-Fort Worth", "primary_state" => "TX" },
+  categories: ["mechanic", "instructor", "gear"]
+)
+Prospector::FetchJob.perform_later(run.id)
+
+# Seed keywords manually (instead of LLM generation)
+Prospector::Keyword.create!(
+  domain: "motorcycle_services",
+  category: "mechanic",
+  keyword: "motorcycle repair shop",
+  source: "manual"
+)
+
+# Approve a candidate programmatically
+candidate = Prospector::Candidate.find(id)
+candidate.approve!  # fires the on_approve callback
+```
+
+### Geography Types
+
+```ruby
+# Metro area (text search)
+Prospector::Geography::MetroArea.new(name: "San Francisco", primary_state: "CA")
+
+# City (text search)
+Prospector::Geography::City.new(city: "Austin", state: "TX")
+
+# Coordinates (nearby search)
+Prospector::Geography::Coordinates.new(lat: 30.267, lng: -97.743, radius_meters: 10_000)
+
+# ZIP code (text search)
+Prospector::Geography::ZipCode.new(zip: "75201")
+
+# Bounding box (nearby search, converted to center + radius)
+Prospector::Geography::BoundingBox.new(ne_lat: 33.0, ne_lng: -96.0, sw_lat: 32.0, sw_lng: -97.0)
+```
+
+## Custom Source Adapters
+
+Prospector ships with Google Places. To add another source:
+
+```ruby
+# lib/my_app/yelp_adapter.rb
+class MyApp::YelpAdapter < Prospector::Sources::Base
+  def self.adapter_key = "yelp"
+
+  def fetch(geography:, keywords:)
+    # Call the Yelp API for each keyword + geography combination.
+    # Return an Array of Prospector::Sources::Result.
+    keywords.flat_map do |keyword|
+      search_yelp(keyword, geography).map do |biz|
+        Prospector::Sources::Result.new(
+          uid: biz["id"],
+          name: biz["name"],
+          formatted_address: biz["location"]["display_address"].join(", "),
+          latitude: biz["coordinates"]["latitude"],
+          longitude: biz["coordinates"]["longitude"],
+          phone_number: biz["phone"],
+          website: biz["url"],
+          description: nil,
+          category: keyword,
+          hours: nil,
+          rating: biz["rating"],
+          rating_count: biz["review_count"],
+          types: biz["categories"].map { |c| c["alias"] },
+          raw: biz
+        )
+      end
+    end
+  end
+end
+```
+
+Register it in your initializer:
+
+```ruby
+Prospector.configure do |config|
+  config.register_source :yelp, MyApp::YelpAdapter
+end
+```
+
+Then select "Yelp" as the source when creating a run.
+
+## Database Tables
+
+Prospector creates four tables, all prefixed with `prospector_`:
+
+| Table | Purpose |
+|-------|---------|
+| `prospector_runs` | Discovery runs with status, geography, and progress counters |
+| `prospector_candidates` | Discovered businesses pending review |
+| `prospector_classification_runs` | Tracks AI reclassification operations |
+| `prospector_keywords` | Stored search keywords per domain and category |
+
+## Architecture
+
+```
+Prospector::Run (state machine)
+  |
+  |-- FetchJob
+  |     |-- Keywords::Generator (DB cache -> LLM fallback)
+  |     |-- Sources::GooglePlaces::Adapter (or custom adapter)
+  |     |-- Pipeline::Orchestrator
+  |     |     |-- Pipeline::Normalizer (address parsing)
+  |     |     |-- Duplicate checking (via config.duplicate_check)
+  |     |     '-- Creates Prospector::Candidate records
+  |     '-- Enqueues ClassifyJob
+  |
+  |-- ClassifyJob
+  |     '-- Classification::Runner
+  |           |-- Calls LLM via ruby_llm
+  |           |-- Stores categories, confidence, reasoning in metadata
+  |           '-- Auto-rejects candidates with no relevant categories
+  |
+  '-- Admin UI
+        |-- Review candidates (approve / reject / restore)
+        |-- Bulk approve
+        '-- Trigger reclassification with different models
+```
+
+## Run States
+
+```
+pending --> running --> classifying --> completed
+               |                          |
+               v                          v
+            failed                    cancelled
+               ^                          ^
+               |__________________________|
+                    (retry / restart)
+```
+
+## Development
+
+The gem ships with a devcontainer for containerized development:
+
+```bash
+# Start the devcontainer
+docker compose -f .devcontainer/compose.yaml up -d --build
+
+# Run tests (inside the container)
+docker exec -w /workspaces/prospector prospector-app-1 bundle exec rake test
+
+# Run a single test file
+docker exec -w /workspaces/prospector prospector-app-1 bundle exec ruby -Itest test/models/run_test.rb
+```
+
+The test suite uses a dummy Rails app (`test/dummy/`) with PostgreSQL.
+
+## License
+
+MIT License. See [MIT-LICENSE](MIT-LICENSE).

data/Rakefile

@@ -0,0 +1,9 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.pattern = "test/**/*_test.rb"
+end
+
+task default: :test

data/app/CLAUDE.md

@@ -0,0 +1,43 @@
+# App Code
+
+## Models (`models/prospector/`)
+
+All models inherit from `Prospector::ApplicationRecord` which sets `table_name_prefix = "prospector_"`.
+
+- `Run` - Central orchestration record. State machine via explicit predicate methods. Associations: has_many candidates, has_many classification_runs. Key methods: `cancel!`, `reset_for_retry!`, `restart!`, `geography`/`geography=`.
+- `Candidate` - Raw scraped business record. Key methods: `approve!` (fires on_approve callback OUTSIDE transaction), `reject!(reason:)`, `restore_to_pending!`, `approvable?`. Metadata stores normalized_data, llm_categories, llm_confidence, llm_reasoning.
+- `ClassificationRun` - Tracks reclassification operations per run.
+- `Keyword` - Stored search keywords. Scoped by domain + category. `keywords_for(domain:, category:)` returns active keyword strings.
+
+## Controllers (`controllers/prospector/`)
+
+All inherit from `Prospector::ApplicationController` which runs `authenticate_prospector_admin!` before_action.
+
+- `RunsController` - index (list runs), show (run detail + candidates), new/create (start a run)
+- `CandidatesController` - show (candidate detail), update (approve/reject/restore with state guards)
+- `RunRetriesController` - create (retry failed run, guards retryable?)
+- `RunRestartsController` - create (restart completed/failed run, guards restartable?)
+- `RunCancellationsController` - create (cancel active run, guards cancellable?)
+- `RunReclassificationsController` - create (trigger reclassification with chosen model)
+- `RunBulkApprovalsController` - create (bulk approve pending candidates, guards completed?/classifying?)
+
+## Jobs (`jobs/prospector/`)
+
+All inherit from `Prospector::ApplicationJob` which uses the configured queue name.
+
+- `FetchJob` - Delegates to `Pipeline::Orchestrator`. Enqueued by RunsController#create.
+- `ClassifyJob` - Delegates to `Classification::Runner`. Enqueued after fetch completes.
+- `BulkApproveJob` - Approves all approvable pending candidates in a run.
+
+## Views (`views/prospector/`)
+
+Self-contained admin UI. Layout at `layouts/prospector.html.erb` with scoped CSS (`prospector/application.css`).
+
+- `runs/index` - List of all runs with status badges
+- `runs/show` - Run detail with stats, action buttons, candidate tabs (pending/approved/rejected)
+- `runs/new` - Form to create a new run (geography type, source, categories)
+- `candidates/show` - Candidate detail with AI classification results
+
+## Assets (`assets/stylesheets/prospector/`)
+
+- `application.css` - Self-contained CSS scoped under `.prospector-ui`. No Tailwind dependency. CSS custom properties for theming.