topical 0.0.1.pre.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: '0559da99a2324dc6a4716e075d1222dfb9049f188e224f0976349aa6cdce5002'
+  data.tar.gz: 03b340c866055159801e476d61866b0b13856966f34b0c0904a49a299fce7ba7
+SHA512:
+  metadata.gz: 535427f8e78814474292ad924dfd367bebc0bf3b796e010e407e8432d1aade54db55599b613255002db0eebfacf6939785699db344cd7d4d19be8eaeb744e804
+  data.tar.gz: 4390654a5a9c7375e10b0174e2a67b4a2e5df7a69176f0761e9a4c4005cdd5b53736d2639a74a1943dc408b2677e2cdf57e1c51a0fcd1886bce8a3db1c83fa27

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.standard.yml

@@ -0,0 +1,3 @@
+# For available configuration options, see:
+#   https://github.com/standardrb/standard
+ruby_version: 3.1

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,132 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes,
+  and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall
+  community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of
+  any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address,
+  without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+[INSERT CONTACT METHOD].
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining
+the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior. No
+interaction with the people involved, including unsolicited interaction with
+those enforcing the Code of Conduct, for a specified period of time. This
+includes avoiding interactions in community spaces as well as external channels
+like social media. Violating these terms may lead to a temporary or permanent
+ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time. No public or
+private interaction with the people involved, including unsolicited interaction
+with those enforcing the Code of Conduct, is allowed during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage],
+version 2.1, available at
+[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by
+[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
+[https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Chris Petersen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,252 @@
+# Topical
+
+Topic modeling for Ruby using modern clustering algorithms. Extract meaningful topics from document embeddings using HDBSCAN clustering and c-TF-IDF term extraction.
+
+## Quick Start
+
+```bash
+# Install the gem
+gem install topical
+
+# Try it out immediately in IRB
+irb
+```
+
+```ruby
+require 'topical'
+
+# Create some sample documents
+documents = [
+  "Ruby is a dynamic programming language with elegant syntax",
+  "Rails is a web framework written in Ruby for building web applications",
+  "Python is great for machine learning and data science applications",
+  "TensorFlow and PyTorch are popular machine learning frameworks in Python",
+  "JavaScript runs in browsers and Node.js for full-stack development",
+  "React and Vue are modern JavaScript frameworks for building UIs",
+  "Machine learning models need training data and validation sets",
+  "Deep learning uses neural networks with multiple layers",
+  "Web development involves HTML, CSS, and JavaScript",
+  "Backend development often uses databases and APIs"
+]
+
+# Create simple mock embeddings (in practice, use real embeddings from red-candle or other embedding models)
+# Here we create 3 distinct clusters based on keywords
+embeddings = documents.map do |doc|
+  text = doc.downcase
+  [
+    text.include?("ruby") || text.include?("rails") ? 1.0 : 0.0,  # Ruby cluster
+    text.include?("python") || text.include?("machine") || text.include?("learning") ? 1.0 : 0.0,  # ML cluster  
+    text.include?("javascript") || text.include?("web") || text.include?("css") ? 1.0 : 0.0,  # Web cluster
+    rand(-0.1..0.1)  # Small random noise
+  ]
+end
+
+# Extract topics
+topics = Topical.extract(
+  embeddings: embeddings,
+  documents: documents,
+  clustering_method: :kmeans,
+  k: 3
+)
+
+# Display results
+topics.each do |topic|
+  puts "\n📌 #{topic.label}"
+  puts "   Documents: #{topic.size}"
+  puts "   Key terms: #{topic.terms.first(5).join(', ')}"
+  puts "   Sample: \"#{topic.documents.first[0..80]}...\""
+end
+```
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'topical'
+
+# Optional but recommended: for generating real embeddings
+gem 'red-candle'  
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install topical
+
+## Real-World Usage with Embeddings
+
+### Using with red-candle (recommended)
+
+```ruby
+require 'topical'
+require 'red-candle'
+
+# Initialize embedding model
+embedder = RedCandle::Embedding.new("sentence-transformers/all-MiniLM-L6-v2")
+
+# Your documents
+documents = [
+  "The Federal Reserve raised interest rates to combat inflation",
+  "Stock markets rallied on positive earnings reports",
+  "New AI breakthrough in natural language processing",
+  "Machine learning transforms healthcare diagnostics",
+  # ... more documents
+]
+
+# Generate embeddings
+embeddings = documents.map { |doc| embedder.embed(doc) }
+
+# Extract topics with HDBSCAN clustering
+engine = Topical::Engine.new(
+  clustering_method: :hdbscan,
+  min_cluster_size: 5,
+  verbose: true
+)
+
+topics = engine.fit(embeddings, documents)
+
+# Analyze results
+topics.each do |topic|
+  puts "\nTopic: #{topic.label}"
+  puts "Size: #{topic.size} documents"
+  puts "Coherence: #{topic.coherence.round(3)}"
+  puts "Top terms: #{topic.terms.first(10).join(', ')}"
+  puts "\nRepresentative documents:"
+  topic.representative_docs(k: 3).each { |doc| puts "  - #{doc[0..100]}..." }
+end
+
+# Check for outliers
+outliers = engine.outliers
+puts "\nOutliers: #{outliers.length} documents"
+```
+
+### Advanced Configuration
+
+```ruby
+# Create engine with custom configuration
+engine = Topical::Engine.new(
+  # Clustering options
+  clustering_method: :hdbscan,    # :hdbscan or :kmeans
+  min_cluster_size: 10,            # Minimum documents per topic (HDBSCAN)
+  min_samples: 5,                  # Core points needed (HDBSCAN)
+  k: 20,                           # Number of topics (K-means only)
+  
+  # Dimensionality reduction
+  reduce_dimensions: true,          # Auto-reduce high-dim embeddings with UMAP
+  n_components: 50,                 # Target dimensions for reduction
+  
+  # Labeling options
+  labeling_method: :hybrid,         # :term_based, :llm_based, or :hybrid
+  llm_provider: nil,               # Optional: custom LLM provider
+  
+  # Other options
+  verbose: true                     # Show progress
+)
+
+# Fit the model
+topics = engine.fit(embeddings, documents, metadata: metadata)
+
+# Save and load models
+engine.save("topic_model.json")
+loaded = Topical::Engine.load("topic_model.json")
+
+# Transform new documents
+new_topics = engine.transform(new_embeddings)
+
+# Get specific topic
+topic = engine.get_topic(0)
+```
+
+### Topic Analysis
+
+```ruby
+# Access topic properties
+topic.id                          # Cluster ID
+topic.label                        # Human-readable label
+topic.terms                        # Top distinctive terms (c-TF-IDF)
+topic.documents                    # All documents in topic
+topic.size                         # Number of documents
+topic.coherence                    # Topic quality score (0-1)
+topic.centroid                     # Topic centroid in embedding space
+
+# Get representative documents
+topic.representative_docs(k: 5)    # 5 most representative docs
+
+# Convert to hash for serialization
+topic.to_h
+
+# Compute metrics across all topics
+diversity = Topical::Metrics.compute_diversity(topics)
+coverage = Topical::Metrics.compute_coverage(topics, total_docs)
+```
+
+## Clustering Methods
+
+### HDBSCAN (Hierarchical Density-Based Clustering)
+- **Pros**: Automatically determines number of topics, identifies outliers, handles varying densities
+- **Cons**: Requires tuning min_cluster_size and min_samples
+- **When to use**: When you don't know the number of topics in advance
+
+### K-means
+- **Pros**: Fast, deterministic with same seed, always assigns all documents
+- **Cons**: Must specify k (number of topics), no outlier detection
+- **When to use**: When you know approximately how many topics to expect
+
+## Term Extraction
+
+Topical uses **c-TF-IDF** (class-based TF-IDF) to find distinctive terms for each topic:
+- Higher scores for terms frequent in topic but rare in other topics
+- Automatically filters stop words
+- Configurable minimum/maximum word lengths
+
+## Topic Labeling Methods
+
+1. **Term-based** (`:term_based`)
+   - Fast, uses top distinctive terms
+   - No external dependencies
+   
+2. **LLM-based** (`:llm_based`)
+   - High quality, contextual labels
+   - Requires red-candle or API provider
+   
+3. **Hybrid** (`:hybrid`)
+   - Best of both: fast with LLM enhancement
+   - Falls back to term-based if LLM unavailable
+
+## Dependencies
+
+- **Required**: `clusterkit` - For HDBSCAN clustering and UMAP dimensionality reduction
+- **Optional**: `red-candle` - For generating embeddings and LLM-powered topic labeling
+
+## Performance Tips
+
+1. **Dimensionality Reduction**: For embeddings with >100 dimensions, enable `reduce_dimensions: true`
+2. **Batch Processing**: Process documents in batches of 1000-5000 for large datasets
+3. **Caching**: Save fitted models with `engine.save()` to avoid recomputation
+4. **Parallel Processing**: Generate embeddings in parallel when possible
+
+## Examples
+
+Check out the `examples/` directory for complete examples:
+- `quick_demo.rb` - Simple demonstration with mock data
+- `news_clustering.rb` - Clustering news articles
+- `customer_feedback.rb` - Analyzing customer feedback topics
+- `research_papers.rb` - Organizing research papers by topic
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt.
+
+To install this gem onto your local machine, run `bundle exec rake install`.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/cpetersen/topical.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "standard/rake"
+
+task default: %i[spec standard]

data/examples/quick_demo.rb

@@ -0,0 +1,118 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'topical'
+
+puts "🎯 Topical Quick Demo"
+puts "=" * 50
+puts
+
+# Create some sample documents about different topics
+documents = [
+  # Ruby/Rails cluster
+  "Ruby is a dynamic programming language with elegant syntax",
+  "Rails is a web framework written in Ruby for building web applications",
+  "Ruby on Rails follows the MVC pattern and convention over configuration",
+  
+  # Python/ML cluster
+  "Python is great for machine learning and data science applications",
+  "TensorFlow and PyTorch are popular machine learning frameworks in Python",
+  "Machine learning models need training data and validation sets",
+  "Deep learning uses neural networks with multiple layers",
+  
+  # JavaScript/Web cluster
+  "JavaScript runs in browsers and Node.js for full-stack development",
+  "React and Vue are modern JavaScript frameworks for building UIs",
+  "Web development involves HTML, CSS, and JavaScript",
+  "Frontend frameworks help build interactive user interfaces",
+  
+  # Database cluster
+  "SQL databases use structured queries to manage relational data",
+  "NoSQL databases like MongoDB store documents in flexible schemas",
+  "Database indexing improves query performance significantly"
+]
+
+puts "📚 Processing #{documents.length} documents..."
+puts
+
+# Create simple mock embeddings based on keywords
+# In real usage, you'd use actual embeddings from red-candle or similar
+embeddings = documents.map do |doc|
+  text = doc.downcase
+  [
+    # Feature engineering based on topic keywords
+    text.include?("ruby") || text.include?("rails") ? 1.0 : 0.0,
+    text.include?("python") || text.include?("machine") || text.include?("learning") || text.include?("neural") ? 1.0 : 0.0,
+    text.include?("javascript") || text.include?("react") || text.include?("vue") || text.include?("web") || text.include?("frontend") ? 1.0 : 0.0,
+    text.include?("database") || text.include?("sql") || text.include?("mongodb") || text.include?("query") ? 1.0 : 0.0,
+    rand(-0.1..0.1),  # Add small random noise
+    rand(-0.1..0.1)
+  ]
+end
+
+# Extract topics using K-means (since we know we have ~4 topic areas)
+puts "🔍 Extracting topics with K-means clustering..."
+topics = Topical.extract(
+  embeddings: embeddings,
+  documents: documents,
+  clustering_method: :kmeans,
+  k: 4,
+  verbose: false
+)
+
+puts "✅ Found #{topics.length} topics!"
+puts
+
+# Display the results
+topics.each_with_index do |topic, i|
+  puts "━" * 50
+  puts "📌 Topic #{i + 1}: #{topic.label}"
+  puts "━" * 50
+  puts "📊 Size: #{topic.size} documents"
+  puts "🔤 Key terms: #{topic.terms.first(6).join(', ')}"
+  puts "📈 Coherence: #{(topic.coherence * 100).round(1)}%"
+  puts
+  puts "📄 Documents in this topic:"
+  topic.documents.each_with_index do |doc, j|
+    preview = doc.length > 60 ? "#{doc[0..60]}..." : doc
+    puts "   #{j + 1}. #{preview}"
+  end
+  puts
+end
+
+# Show topic diversity
+diversity = Topical::Metrics.compute_diversity(topics)
+puts "━" * 50
+puts "📊 Overall topic diversity: #{(diversity * 100).round(1)}%"
+puts "💡 Higher diversity means topics are more distinct from each other"
+puts
+
+# Test HDBSCAN clustering (density-based, finds optimal number of clusters)
+puts "━" * 50
+puts "🔍 Now trying HDBSCAN clustering (automatic topic detection)..."
+puts
+
+engine = Topical::Engine.new(
+  clustering_method: :hdbscan,
+  min_cluster_size: 3,  # Minimum 3 docs per topic
+  min_samples: 2
+)
+
+hdbscan_topics = engine.fit(embeddings, documents)
+outliers = engine.outliers
+
+puts "✅ HDBSCAN found #{hdbscan_topics.length} topics"
+puts "🔸 Outliers: #{outliers.length} documents"
+
+if outliers.any?
+  puts "\nDocuments marked as outliers (don't fit well in any topic):"
+  outliers.each { |doc| puts "  - #{doc[0..60]}..." }
+end
+
+puts
+puts "━" * 50
+puts "🎉 Demo complete! Try it with your own documents and real embeddings."
+puts
+puts "💡 Tip: Install red-candle to generate real embeddings:"
+puts "   gem install red-candle"
+puts "   Then use: RedCandle::Embedding.new('model-name').embed(text)"

data/examples/verify_migration.rb

@@ -0,0 +1,108 @@
+#!/usr/bin/env ruby
+
+require 'bundler/setup'
+require 'topical'
+
+puts "=== Topical Migration Verification ==="
+puts "Version: #{Topical::VERSION}"
+puts "LLM Available: #{Topical.llm_available?}"
+puts
+
+# Create test data
+embeddings = []
+documents = []
+
+# Tech cluster
+10.times do |i|
+  embeddings << [1.0 + rand(-0.1..0.1), rand(-0.1..0.1), rand(-0.1..0.1)]
+  documents << "Python programming #{i} with machine learning and neural networks"
+end
+
+# Business cluster
+10.times do |i|
+  embeddings << [rand(-0.1..0.1), 1.0 + rand(-0.1..0.1), rand(-0.1..0.1)]
+  documents << "Financial markets #{i} with investment portfolio and trading strategies"
+end
+
+# Science cluster
+10.times do |i|
+  embeddings << [rand(-0.1..0.1), rand(-0.1..0.1), 1.0 + rand(-0.1..0.1)]
+  documents << "Scientific research #{i} with experimental methodology and peer review"
+end
+
+puts "Testing different clustering methods:"
+puts
+
+# Test HDBSCAN
+puts "1. HDBSCAN Clustering:"
+engine = Topical::Engine.new(
+  clustering_method: :hdbscan,
+  min_cluster_size: 5,
+  verbose: true
+)
+topics = engine.fit(embeddings, documents)
+puts "  Found #{topics.length} topics"
+topics.each do |topic|
+  puts "    Topic #{topic.id}: #{topic.label} (#{topic.size} docs)"
+end
+puts
+
+# Test K-means
+puts "2. K-means Clustering:"
+engine = Topical::Engine.new(
+  clustering_method: :kmeans,
+  k: 3,
+  verbose: false
+)
+topics = engine.fit(embeddings, documents)
+puts "  Found #{topics.length} topics"
+topics.each do |topic|
+  puts "    Topic #{topic.id}: #{topic.label} (#{topic.size} docs)"
+end
+puts
+
+# Test convenience method
+puts "3. Convenience API:"
+topics = Topical.extract(
+  embeddings: embeddings,
+  documents: documents,
+  clustering_method: :kmeans,
+  k: 3
+)
+puts "  Found #{topics.length} topics"
+puts
+
+# Test advanced features
+puts "4. Advanced Features:"
+topic = topics.first
+puts "  Representative docs: #{topic.representative_docs(k: 2).length}"
+puts "  Coherence: #{topic.coherence.round(3)}"
+puts "  Centroid dimensions: #{topic.centroid.length}"
+puts
+
+# Test metrics
+puts "5. Metrics Module:"
+diversity = Topical::Metrics.compute_diversity(topics)
+coverage = Topical::Metrics.compute_coverage(topics, documents.length)
+puts "  Topic diversity: #{diversity.round(3)}"
+puts "  Document coverage: #{coverage.round(3)}"
+puts
+
+# Test persistence
+puts "6. Model Persistence:"
+model_path = "/tmp/topical_test_model.json"
+engine.save(model_path)
+puts "  Model saved to #{model_path}"
+
+loaded_engine = Topical::Engine.load(model_path)
+puts "  Model loaded successfully"
+puts "  Loaded topics: #{loaded_engine.topics.length}"
+puts
+
+# Test outlier detection
+puts "7. Outlier Detection:"
+outliers = engine.outliers
+puts "  Outliers: #{outliers.length}"
+puts
+
+puts "=== All tests passed! Migration successful. ===" 

data/lib/topical/clustering/adapter.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Topical
+  module Clustering
+    # Base adapter class for clustering algorithms
+    class Adapter
+      def fit_predict(embeddings)
+        raise NotImplementedError, "Subclasses must implement fit_predict"
+      end
+      
+      def fit(embeddings)
+        raise NotImplementedError, "Subclasses must implement fit"
+      end
+      
+      def predict(embeddings)
+        raise NotImplementedError, "Subclasses must implement predict"
+      end
+      
+      # Number of clusters found (excluding noise)
+      def n_clusters
+        @n_clusters || 0
+      end
+      
+      # Number of noise points (labeled as -1)
+      def n_noise_points
+        @n_noise_points || 0
+      end
+    end
+  end
+end