topical 0.0.1.pre.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: '0559da99a2324dc6a4716e075d1222dfb9049f188e224f0976349aa6cdce5002'
4
+ data.tar.gz: 03b340c866055159801e476d61866b0b13856966f34b0c0904a49a299fce7ba7
5
+ SHA512:
6
+ metadata.gz: 535427f8e78814474292ad924dfd367bebc0bf3b796e010e407e8432d1aade54db55599b613255002db0eebfacf6939785699db344cd7d4d19be8eaeb744e804
7
+ data.tar.gz: 4390654a5a9c7375e10b0174e2a67b4a2e5df7a69176f0761e9a4c4005cdd5b53736d2639a74a1943dc408b2677e2cdf57e1c51a0fcd1886bce8a3db1c83fa27
data/.rspec ADDED
@@ -0,0 +1,3 @@
1
+ --format documentation
2
+ --color
3
+ --require spec_helper
data/.standard.yml ADDED
@@ -0,0 +1,3 @@
1
+ # For available configuration options, see:
2
+ # https://github.com/standardrb/standard
3
+ ruby_version: 3.1
@@ -0,0 +1,132 @@
1
+ # Contributor Covenant Code of Conduct
2
+
3
+ ## Our Pledge
4
+
5
+ We as members, contributors, and leaders pledge to make participation in our
6
+ community a harassment-free experience for everyone, regardless of age, body
7
+ size, visible or invisible disability, ethnicity, sex characteristics, gender
8
+ identity and expression, level of experience, education, socio-economic status,
9
+ nationality, personal appearance, race, caste, color, religion, or sexual
10
+ identity and orientation.
11
+
12
+ We pledge to act and interact in ways that contribute to an open, welcoming,
13
+ diverse, inclusive, and healthy community.
14
+
15
+ ## Our Standards
16
+
17
+ Examples of behavior that contributes to a positive environment for our
18
+ community include:
19
+
20
+ * Demonstrating empathy and kindness toward other people
21
+ * Being respectful of differing opinions, viewpoints, and experiences
22
+ * Giving and gracefully accepting constructive feedback
23
+ * Accepting responsibility and apologizing to those affected by our mistakes,
24
+ and learning from the experience
25
+ * Focusing on what is best not just for us as individuals, but for the overall
26
+ community
27
+
28
+ Examples of unacceptable behavior include:
29
+
30
+ * The use of sexualized language or imagery, and sexual attention or advances of
31
+ any kind
32
+ * Trolling, insulting or derogatory comments, and personal or political attacks
33
+ * Public or private harassment
34
+ * Publishing others' private information, such as a physical or email address,
35
+ without their explicit permission
36
+ * Other conduct which could reasonably be considered inappropriate in a
37
+ professional setting
38
+
39
+ ## Enforcement Responsibilities
40
+
41
+ Community leaders are responsible for clarifying and enforcing our standards of
42
+ acceptable behavior and will take appropriate and fair corrective action in
43
+ response to any behavior that they deem inappropriate, threatening, offensive,
44
+ or harmful.
45
+
46
+ Community leaders have the right and responsibility to remove, edit, or reject
47
+ comments, commits, code, wiki edits, issues, and other contributions that are
48
+ not aligned to this Code of Conduct, and will communicate reasons for moderation
49
+ decisions when appropriate.
50
+
51
+ ## Scope
52
+
53
+ This Code of Conduct applies within all community spaces, and also applies when
54
+ an individual is officially representing the community in public spaces.
55
+ Examples of representing our community include using an official email address,
56
+ posting via an official social media account, or acting as an appointed
57
+ representative at an online or offline event.
58
+
59
+ ## Enforcement
60
+
61
+ Instances of abusive, harassing, or otherwise unacceptable behavior may be
62
+ reported to the community leaders responsible for enforcement at
63
+ [INSERT CONTACT METHOD].
64
+ All complaints will be reviewed and investigated promptly and fairly.
65
+
66
+ All community leaders are obligated to respect the privacy and security of the
67
+ reporter of any incident.
68
+
69
+ ## Enforcement Guidelines
70
+
71
+ Community leaders will follow these Community Impact Guidelines in determining
72
+ the consequences for any action they deem in violation of this Code of Conduct:
73
+
74
+ ### 1. Correction
75
+
76
+ **Community Impact**: Use of inappropriate language or other behavior deemed
77
+ unprofessional or unwelcome in the community.
78
+
79
+ **Consequence**: A private, written warning from community leaders, providing
80
+ clarity around the nature of the violation and an explanation of why the
81
+ behavior was inappropriate. A public apology may be requested.
82
+
83
+ ### 2. Warning
84
+
85
+ **Community Impact**: A violation through a single incident or series of
86
+ actions.
87
+
88
+ **Consequence**: A warning with consequences for continued behavior. No
89
+ interaction with the people involved, including unsolicited interaction with
90
+ those enforcing the Code of Conduct, for a specified period of time. This
91
+ includes avoiding interactions in community spaces as well as external channels
92
+ like social media. Violating these terms may lead to a temporary or permanent
93
+ ban.
94
+
95
+ ### 3. Temporary Ban
96
+
97
+ **Community Impact**: A serious violation of community standards, including
98
+ sustained inappropriate behavior.
99
+
100
+ **Consequence**: A temporary ban from any sort of interaction or public
101
+ communication with the community for a specified period of time. No public or
102
+ private interaction with the people involved, including unsolicited interaction
103
+ with those enforcing the Code of Conduct, is allowed during this period.
104
+ Violating these terms may lead to a permanent ban.
105
+
106
+ ### 4. Permanent Ban
107
+
108
+ **Community Impact**: Demonstrating a pattern of violation of community
109
+ standards, including sustained inappropriate behavior, harassment of an
110
+ individual, or aggression toward or disparagement of classes of individuals.
111
+
112
+ **Consequence**: A permanent ban from any sort of public interaction within the
113
+ community.
114
+
115
+ ## Attribution
116
+
117
+ This Code of Conduct is adapted from the [Contributor Covenant][homepage],
118
+ version 2.1, available at
119
+ [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
120
+
121
+ Community Impact Guidelines were inspired by
122
+ [Mozilla's code of conduct enforcement ladder][Mozilla CoC].
123
+
124
+ For answers to common questions about this code of conduct, see the FAQ at
125
+ [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
126
+ [https://www.contributor-covenant.org/translations][translations].
127
+
128
+ [homepage]: https://www.contributor-covenant.org
129
+ [v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
130
+ [Mozilla CoC]: https://github.com/mozilla/diversity
131
+ [FAQ]: https://www.contributor-covenant.org/faq
132
+ [translations]: https://www.contributor-covenant.org/translations
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2025 Chris Petersen
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,252 @@
1
+ # Topical
2
+
3
+ Topic modeling for Ruby using modern clustering algorithms. Extract meaningful topics from document embeddings using HDBSCAN clustering and c-TF-IDF term extraction.
4
+
5
+ ## Quick Start
6
+
7
+ ```bash
8
+ # Install the gem
9
+ gem install topical
10
+
11
+ # Try it out immediately in IRB
12
+ irb
13
+ ```
14
+
15
+ ```ruby
16
+ require 'topical'
17
+
18
+ # Create some sample documents
19
+ documents = [
20
+ "Ruby is a dynamic programming language with elegant syntax",
21
+ "Rails is a web framework written in Ruby for building web applications",
22
+ "Python is great for machine learning and data science applications",
23
+ "TensorFlow and PyTorch are popular machine learning frameworks in Python",
24
+ "JavaScript runs in browsers and Node.js for full-stack development",
25
+ "React and Vue are modern JavaScript frameworks for building UIs",
26
+ "Machine learning models need training data and validation sets",
27
+ "Deep learning uses neural networks with multiple layers",
28
+ "Web development involves HTML, CSS, and JavaScript",
29
+ "Backend development often uses databases and APIs"
30
+ ]
31
+
32
+ # Create simple mock embeddings (in practice, use real embeddings from red-candle or other embedding models)
33
+ # Here we create 3 distinct clusters based on keywords
34
+ embeddings = documents.map do |doc|
35
+ text = doc.downcase
36
+ [
37
+ text.include?("ruby") || text.include?("rails") ? 1.0 : 0.0, # Ruby cluster
38
+ text.include?("python") || text.include?("machine") || text.include?("learning") ? 1.0 : 0.0, # ML cluster
39
+ text.include?("javascript") || text.include?("web") || text.include?("css") ? 1.0 : 0.0, # Web cluster
40
+ rand(-0.1..0.1) # Small random noise
41
+ ]
42
+ end
43
+
44
+ # Extract topics
45
+ topics = Topical.extract(
46
+ embeddings: embeddings,
47
+ documents: documents,
48
+ clustering_method: :kmeans,
49
+ k: 3
50
+ )
51
+
52
+ # Display results
53
+ topics.each do |topic|
54
+ puts "\n📌 #{topic.label}"
55
+ puts " Documents: #{topic.size}"
56
+ puts " Key terms: #{topic.terms.first(5).join(', ')}"
57
+ puts " Sample: \"#{topic.documents.first[0..80]}...\""
58
+ end
59
+ ```
60
+
61
+ ## Installation
62
+
63
+ Add this line to your application's Gemfile:
64
+
65
+ ```ruby
66
+ gem 'topical'
67
+
68
+ # Optional but recommended: for generating real embeddings
69
+ gem 'red-candle'
70
+ ```
71
+
72
+ And then execute:
73
+
74
+ $ bundle install
75
+
76
+ Or install it yourself as:
77
+
78
+ $ gem install topical
79
+
80
+ ## Real-World Usage with Embeddings
81
+
82
+ ### Using with red-candle (recommended)
83
+
84
+ ```ruby
85
+ require 'topical'
86
+ require 'red-candle'
87
+
88
+ # Initialize embedding model
89
+ embedder = RedCandle::Embedding.new("sentence-transformers/all-MiniLM-L6-v2")
90
+
91
+ # Your documents
92
+ documents = [
93
+ "The Federal Reserve raised interest rates to combat inflation",
94
+ "Stock markets rallied on positive earnings reports",
95
+ "New AI breakthrough in natural language processing",
96
+ "Machine learning transforms healthcare diagnostics",
97
+ # ... more documents
98
+ ]
99
+
100
+ # Generate embeddings
101
+ embeddings = documents.map { |doc| embedder.embed(doc) }
102
+
103
+ # Extract topics with HDBSCAN clustering
104
+ engine = Topical::Engine.new(
105
+ clustering_method: :hdbscan,
106
+ min_cluster_size: 5,
107
+ verbose: true
108
+ )
109
+
110
+ topics = engine.fit(embeddings, documents)
111
+
112
+ # Analyze results
113
+ topics.each do |topic|
114
+ puts "\nTopic: #{topic.label}"
115
+ puts "Size: #{topic.size} documents"
116
+ puts "Coherence: #{topic.coherence.round(3)}"
117
+ puts "Top terms: #{topic.terms.first(10).join(', ')}"
118
+ puts "\nRepresentative documents:"
119
+ topic.representative_docs(k: 3).each { |doc| puts " - #{doc[0..100]}..." }
120
+ end
121
+
122
+ # Check for outliers
123
+ outliers = engine.outliers
124
+ puts "\nOutliers: #{outliers.length} documents"
125
+ ```
126
+
127
+ ### Advanced Configuration
128
+
129
+ ```ruby
130
+ # Create engine with custom configuration
131
+ engine = Topical::Engine.new(
132
+ # Clustering options
133
+ clustering_method: :hdbscan, # :hdbscan or :kmeans
134
+ min_cluster_size: 10, # Minimum documents per topic (HDBSCAN)
135
+ min_samples: 5, # Core points needed (HDBSCAN)
136
+ k: 20, # Number of topics (K-means only)
137
+
138
+ # Dimensionality reduction
139
+ reduce_dimensions: true, # Auto-reduce high-dim embeddings with UMAP
140
+ n_components: 50, # Target dimensions for reduction
141
+
142
+ # Labeling options
143
+ labeling_method: :hybrid, # :term_based, :llm_based, or :hybrid
144
+ llm_provider: nil, # Optional: custom LLM provider
145
+
146
+ # Other options
147
+ verbose: true # Show progress
148
+ )
149
+
150
+ # Fit the model
151
+ topics = engine.fit(embeddings, documents, metadata: metadata)
152
+
153
+ # Save and load models
154
+ engine.save("topic_model.json")
155
+ loaded = Topical::Engine.load("topic_model.json")
156
+
157
+ # Transform new documents
158
+ new_topics = engine.transform(new_embeddings)
159
+
160
+ # Get specific topic
161
+ topic = engine.get_topic(0)
162
+ ```
163
+
164
+ ### Topic Analysis
165
+
166
+ ```ruby
167
+ # Access topic properties
168
+ topic.id # Cluster ID
169
+ topic.label # Human-readable label
170
+ topic.terms # Top distinctive terms (c-TF-IDF)
171
+ topic.documents # All documents in topic
172
+ topic.size # Number of documents
173
+ topic.coherence # Topic quality score (0-1)
174
+ topic.centroid # Topic centroid in embedding space
175
+
176
+ # Get representative documents
177
+ topic.representative_docs(k: 5) # 5 most representative docs
178
+
179
+ # Convert to hash for serialization
180
+ topic.to_h
181
+
182
+ # Compute metrics across all topics
183
+ diversity = Topical::Metrics.compute_diversity(topics)
184
+ coverage = Topical::Metrics.compute_coverage(topics, total_docs)
185
+ ```
186
+
187
+ ## Clustering Methods
188
+
189
+ ### HDBSCAN (Hierarchical Density-Based Clustering)
190
+ - **Pros**: Automatically determines number of topics, identifies outliers, handles varying densities
191
+ - **Cons**: Requires tuning min_cluster_size and min_samples
192
+ - **When to use**: When you don't know the number of topics in advance
193
+
194
+ ### K-means
195
+ - **Pros**: Fast, deterministic with same seed, always assigns all documents
196
+ - **Cons**: Must specify k (number of topics), no outlier detection
197
+ - **When to use**: When you know approximately how many topics to expect
198
+
199
+ ## Term Extraction
200
+
201
+ Topical uses **c-TF-IDF** (class-based TF-IDF) to find distinctive terms for each topic:
202
+ - Higher scores for terms frequent in topic but rare in other topics
203
+ - Automatically filters stop words
204
+ - Configurable minimum/maximum word lengths
205
+
206
+ ## Topic Labeling Methods
207
+
208
+ 1. **Term-based** (`:term_based`)
209
+ - Fast, uses top distinctive terms
210
+ - No external dependencies
211
+
212
+ 2. **LLM-based** (`:llm_based`)
213
+ - High quality, contextual labels
214
+ - Requires red-candle or API provider
215
+
216
+ 3. **Hybrid** (`:hybrid`)
217
+ - Best of both: fast with LLM enhancement
218
+ - Falls back to term-based if LLM unavailable
219
+
220
+ ## Dependencies
221
+
222
+ - **Required**: `clusterkit` - For HDBSCAN clustering and UMAP dimensionality reduction
223
+ - **Optional**: `red-candle` - For generating embeddings and LLM-powered topic labeling
224
+
225
+ ## Performance Tips
226
+
227
+ 1. **Dimensionality Reduction**: For embeddings with >100 dimensions, enable `reduce_dimensions: true`
228
+ 2. **Batch Processing**: Process documents in batches of 1000-5000 for large datasets
229
+ 3. **Caching**: Save fitted models with `engine.save()` to avoid recomputation
230
+ 4. **Parallel Processing**: Generate embeddings in parallel when possible
231
+
232
+ ## Examples
233
+
234
+ Check out the `examples/` directory for complete examples:
235
+ - `quick_demo.rb` - Simple demonstration with mock data
236
+ - `news_clustering.rb` - Clustering news articles
237
+ - `customer_feedback.rb` - Analyzing customer feedback topics
238
+ - `research_papers.rb` - Organizing research papers by topic
239
+
240
+ ## Development
241
+
242
+ After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt.
243
+
244
+ To install this gem onto your local machine, run `bundle exec rake install`.
245
+
246
+ ## Contributing
247
+
248
+ Bug reports and pull requests are welcome on GitHub at https://github.com/cpetersen/topical.
249
+
250
+ ## License
251
+
252
+ The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
data/Rakefile ADDED
@@ -0,0 +1,10 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "bundler/gem_tasks"
4
+ require "rspec/core/rake_task"
5
+
6
+ RSpec::Core::RakeTask.new(:spec)
7
+
8
+ require "standard/rake"
9
+
10
+ task default: %i[spec standard]
@@ -0,0 +1,118 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'bundler/setup'
4
+ require 'topical'
5
+
6
+ puts "🎯 Topical Quick Demo"
7
+ puts "=" * 50
8
+ puts
9
+
10
+ # Create some sample documents about different topics
11
+ documents = [
12
+ # Ruby/Rails cluster
13
+ "Ruby is a dynamic programming language with elegant syntax",
14
+ "Rails is a web framework written in Ruby for building web applications",
15
+ "Ruby on Rails follows the MVC pattern and convention over configuration",
16
+
17
+ # Python/ML cluster
18
+ "Python is great for machine learning and data science applications",
19
+ "TensorFlow and PyTorch are popular machine learning frameworks in Python",
20
+ "Machine learning models need training data and validation sets",
21
+ "Deep learning uses neural networks with multiple layers",
22
+
23
+ # JavaScript/Web cluster
24
+ "JavaScript runs in browsers and Node.js for full-stack development",
25
+ "React and Vue are modern JavaScript frameworks for building UIs",
26
+ "Web development involves HTML, CSS, and JavaScript",
27
+ "Frontend frameworks help build interactive user interfaces",
28
+
29
+ # Database cluster
30
+ "SQL databases use structured queries to manage relational data",
31
+ "NoSQL databases like MongoDB store documents in flexible schemas",
32
+ "Database indexing improves query performance significantly"
33
+ ]
34
+
35
+ puts "📚 Processing #{documents.length} documents..."
36
+ puts
37
+
38
+ # Create simple mock embeddings based on keywords
39
+ # In real usage, you'd use actual embeddings from red-candle or similar
40
+ embeddings = documents.map do |doc|
41
+ text = doc.downcase
42
+ [
43
+ # Feature engineering based on topic keywords
44
+ text.include?("ruby") || text.include?("rails") ? 1.0 : 0.0,
45
+ text.include?("python") || text.include?("machine") || text.include?("learning") || text.include?("neural") ? 1.0 : 0.0,
46
+ text.include?("javascript") || text.include?("react") || text.include?("vue") || text.include?("web") || text.include?("frontend") ? 1.0 : 0.0,
47
+ text.include?("database") || text.include?("sql") || text.include?("mongodb") || text.include?("query") ? 1.0 : 0.0,
48
+ rand(-0.1..0.1), # Add small random noise
49
+ rand(-0.1..0.1)
50
+ ]
51
+ end
52
+
53
+ # Extract topics using K-means (since we know we have ~4 topic areas)
54
+ puts "🔍 Extracting topics with K-means clustering..."
55
+ topics = Topical.extract(
56
+ embeddings: embeddings,
57
+ documents: documents,
58
+ clustering_method: :kmeans,
59
+ k: 4,
60
+ verbose: false
61
+ )
62
+
63
+ puts "✅ Found #{topics.length} topics!"
64
+ puts
65
+
66
+ # Display the results
67
+ topics.each_with_index do |topic, i|
68
+ puts "━" * 50
69
+ puts "📌 Topic #{i + 1}: #{topic.label}"
70
+ puts "━" * 50
71
+ puts "📊 Size: #{topic.size} documents"
72
+ puts "🔤 Key terms: #{topic.terms.first(6).join(', ')}"
73
+ puts "📈 Coherence: #{(topic.coherence * 100).round(1)}%"
74
+ puts
75
+ puts "📄 Documents in this topic:"
76
+ topic.documents.each_with_index do |doc, j|
77
+ preview = doc.length > 60 ? "#{doc[0..60]}..." : doc
78
+ puts " #{j + 1}. #{preview}"
79
+ end
80
+ puts
81
+ end
82
+
83
+ # Show topic diversity
84
+ diversity = Topical::Metrics.compute_diversity(topics)
85
+ puts "━" * 50
86
+ puts "📊 Overall topic diversity: #{(diversity * 100).round(1)}%"
87
+ puts "💡 Higher diversity means topics are more distinct from each other"
88
+ puts
89
+
90
+ # Test HDBSCAN clustering (density-based, finds optimal number of clusters)
91
+ puts "━" * 50
92
+ puts "🔍 Now trying HDBSCAN clustering (automatic topic detection)..."
93
+ puts
94
+
95
+ engine = Topical::Engine.new(
96
+ clustering_method: :hdbscan,
97
+ min_cluster_size: 3, # Minimum 3 docs per topic
98
+ min_samples: 2
99
+ )
100
+
101
+ hdbscan_topics = engine.fit(embeddings, documents)
102
+ outliers = engine.outliers
103
+
104
+ puts "✅ HDBSCAN found #{hdbscan_topics.length} topics"
105
+ puts "🔸 Outliers: #{outliers.length} documents"
106
+
107
+ if outliers.any?
108
+ puts "\nDocuments marked as outliers (don't fit well in any topic):"
109
+ outliers.each { |doc| puts " - #{doc[0..60]}..." }
110
+ end
111
+
112
+ puts
113
+ puts "━" * 50
114
+ puts "🎉 Demo complete! Try it with your own documents and real embeddings."
115
+ puts
116
+ puts "💡 Tip: Install red-candle to generate real embeddings:"
117
+ puts " gem install red-candle"
118
+ puts " Then use: RedCandle::Embedding.new('model-name').embed(text)"
@@ -0,0 +1,108 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'bundler/setup'
4
+ require 'topical'
5
+
6
+ puts "=== Topical Migration Verification ==="
7
+ puts "Version: #{Topical::VERSION}"
8
+ puts "LLM Available: #{Topical.llm_available?}"
9
+ puts
10
+
11
+ # Create test data
12
+ embeddings = []
13
+ documents = []
14
+
15
+ # Tech cluster
16
+ 10.times do |i|
17
+ embeddings << [1.0 + rand(-0.1..0.1), rand(-0.1..0.1), rand(-0.1..0.1)]
18
+ documents << "Python programming #{i} with machine learning and neural networks"
19
+ end
20
+
21
+ # Business cluster
22
+ 10.times do |i|
23
+ embeddings << [rand(-0.1..0.1), 1.0 + rand(-0.1..0.1), rand(-0.1..0.1)]
24
+ documents << "Financial markets #{i} with investment portfolio and trading strategies"
25
+ end
26
+
27
+ # Science cluster
28
+ 10.times do |i|
29
+ embeddings << [rand(-0.1..0.1), rand(-0.1..0.1), 1.0 + rand(-0.1..0.1)]
30
+ documents << "Scientific research #{i} with experimental methodology and peer review"
31
+ end
32
+
33
+ puts "Testing different clustering methods:"
34
+ puts
35
+
36
+ # Test HDBSCAN
37
+ puts "1. HDBSCAN Clustering:"
38
+ engine = Topical::Engine.new(
39
+ clustering_method: :hdbscan,
40
+ min_cluster_size: 5,
41
+ verbose: true
42
+ )
43
+ topics = engine.fit(embeddings, documents)
44
+ puts " Found #{topics.length} topics"
45
+ topics.each do |topic|
46
+ puts " Topic #{topic.id}: #{topic.label} (#{topic.size} docs)"
47
+ end
48
+ puts
49
+
50
+ # Test K-means
51
+ puts "2. K-means Clustering:"
52
+ engine = Topical::Engine.new(
53
+ clustering_method: :kmeans,
54
+ k: 3,
55
+ verbose: false
56
+ )
57
+ topics = engine.fit(embeddings, documents)
58
+ puts " Found #{topics.length} topics"
59
+ topics.each do |topic|
60
+ puts " Topic #{topic.id}: #{topic.label} (#{topic.size} docs)"
61
+ end
62
+ puts
63
+
64
+ # Test convenience method
65
+ puts "3. Convenience API:"
66
+ topics = Topical.extract(
67
+ embeddings: embeddings,
68
+ documents: documents,
69
+ clustering_method: :kmeans,
70
+ k: 3
71
+ )
72
+ puts " Found #{topics.length} topics"
73
+ puts
74
+
75
+ # Test advanced features
76
+ puts "4. Advanced Features:"
77
+ topic = topics.first
78
+ puts " Representative docs: #{topic.representative_docs(k: 2).length}"
79
+ puts " Coherence: #{topic.coherence.round(3)}"
80
+ puts " Centroid dimensions: #{topic.centroid.length}"
81
+ puts
82
+
83
+ # Test metrics
84
+ puts "5. Metrics Module:"
85
+ diversity = Topical::Metrics.compute_diversity(topics)
86
+ coverage = Topical::Metrics.compute_coverage(topics, documents.length)
87
+ puts " Topic diversity: #{diversity.round(3)}"
88
+ puts " Document coverage: #{coverage.round(3)}"
89
+ puts
90
+
91
+ # Test persistence
92
+ puts "6. Model Persistence:"
93
+ model_path = "/tmp/topical_test_model.json"
94
+ engine.save(model_path)
95
+ puts " Model saved to #{model_path}"
96
+
97
+ loaded_engine = Topical::Engine.load(model_path)
98
+ puts " Model loaded successfully"
99
+ puts " Loaded topics: #{loaded_engine.topics.length}"
100
+ puts
101
+
102
+ # Test outlier detection
103
+ puts "7. Outlier Detection:"
104
+ outliers = engine.outliers
105
+ puts " Outliers: #{outliers.length}"
106
+ puts
107
+
108
+ puts "=== All tests passed! Migration successful. ==="
@@ -0,0 +1,30 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Topical
4
+ module Clustering
5
+ # Base adapter class for clustering algorithms
6
+ class Adapter
7
+ def fit_predict(embeddings)
8
+ raise NotImplementedError, "Subclasses must implement fit_predict"
9
+ end
10
+
11
+ def fit(embeddings)
12
+ raise NotImplementedError, "Subclasses must implement fit"
13
+ end
14
+
15
+ def predict(embeddings)
16
+ raise NotImplementedError, "Subclasses must implement predict"
17
+ end
18
+
19
+ # Number of clusters found (excluding noise)
20
+ def n_clusters
21
+ @n_clusters || 0
22
+ end
23
+
24
+ # Number of noise points (labeled as -1)
25
+ def n_noise_points
26
+ @n_noise_points || 0
27
+ end
28
+ end
29
+ end
30
+ end