broadlistening 0.7.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 23f5c91f6f1458def1f7d9a8f48ffe18e5065298020042ea83c8ee4b567e747e
+  data.tar.gz: de2380dc80b27514a6831b6e8bb84cef37562e824a42e4e57b55cd07e4e3da71
+SHA512:
+  metadata.gz: 6fc398f25fc7a5dc7935958fc5ff100234d8a52a312b45c6a2f8f77b0b0210a9c89e9186cf3139dd25652834d4d249cbb16ec225e753ea8289f9882f150b7ffc
+  data.tar.gz: 7d4ef0cfceae1f7a781c6d8e03e2bcd2b649d7ffc3c41ce04bc47fc1cdd25689f6a63604b01f88b33f163049b95a0cee869feeb3397834fe8a8a22e268ef22ff

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.rubocop.yml

@@ -0,0 +1,3 @@
+inherit_gem:
+  rubocop-rails-omakase: rubocop.yml
+

data/CHANGELOG.md

@@ -0,0 +1,40 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.7.0] - 2024-11-30
+
+### Added
+- Multi-provider support for LLM
+  - OpenAI (default)
+  - Azure OpenAI
+  - Google Gemini
+  - OpenRouter
+  - Local LLM (Ollama)
+- CLI class compatible with Python kouchou-ai `hierarchical_main.py`
+  - `broadlistening CONFIG [options]` command
+  - `-f, --force` option to force re-run all steps
+  - `-o, --only STEP` option to run only specified step
+  - `--skip-interaction` option to skip confirmation prompt
+  - Auto-generates output directory from config filename (e.g., `config/report.json` → `outputs/report/`)
+- Config class now supports `input`, `question`, `name`, and `intro` fields for Python compatibility
+
+### Changed
+- Extracted Provider class to separate LLM provider configuration from Config
+- Removed Services namespace, moved classes directly to Broadlistening module
+
+### Fixed
+- Changed PROVIDERS hash keys from strings to symbols for consistency
+- Use single worker in notification tests for deterministic behavior
+
+## [0.1.0] - 2024-11-30
+
+### Added
+- Initial implementation of Broadlistening pipeline
+- 7-step pipeline: extraction, embedding, clustering, initial_labelling, merge_labelling, overview, aggregation
+- Incremental execution support with status tracking
+- Output format compatible with Kouchou-AI Python implementation
+- ActiveSupport::Notifications for pipeline events

data/CLAUDE.md

@@ -0,0 +1,112 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Overview
+
+Broadlistening is a Ruby gem that implements the broadlistening pipeline for clustering and analyzing public comments using LLM. It is a Ruby port of the Kouchou-AI (広聴AI) Python implementation, designed for use in Rails applications and other Ruby environments.
+
+## Development Commands
+
+```bash
+# Setup
+bin/setup
+
+# Run all tests
+bundle exec rspec
+
+# Run a single test file
+bundle exec rspec spec/pipeline_spec.rb
+
+# Run compatibility tests only
+bundle exec rspec spec/compatibility/
+
+# Run a specific test by line number
+bundle exec rspec spec/pipeline_spec.rb:42
+
+# Linting
+bundle exec rubocop
+
+# Auto-fix lint issues
+bundle exec rubocop -A
+
+# Interactive console
+bin/console
+```
+
+### Compatibility Tasks
+
+```bash
+# Validate Python output structure against schema
+bundle exec rake compatibility:validate_python
+
+# Compare Python and Ruby outputs
+bundle exec rake "compatibility:compare[python_output.json,ruby_output.json]"
+```
+
+## Architecture
+
+### Pipeline Flow
+
+The pipeline processes comments through 7 sequential steps:
+
+1. **Extraction** - Extract opinions from comments using LLM
+2. **Embedding** - Vectorize extracted opinions using OpenAI embeddings
+3. **Clustering** - UMAP dimensionality reduction + KMeans + hierarchical clustering
+4. **Initial Labelling** - LLM-based labeling for each cluster
+5. **Merge Labelling** - Hierarchical label integration
+6. **Overview** - LLM-generated summary of all clusters
+7. **Aggregation** - Assemble final JSON output
+
+### Key Components
+
+- **Pipeline** (`lib/broadlistening/pipeline.rb`) - Orchestrates step execution, handles incremental execution
+- **Context** (`lib/broadlistening/context.rb`) - Manages all data flowing through pipeline, supports load/save for incremental execution
+- **Config** (`lib/broadlistening/config.rb`) - Configuration management, compatible with Python config.json format
+- **SpecLoader** (`lib/broadlistening/spec_loader.rb`) - Loads step specifications from Python's hierarchical_specs.json
+- **Planner** (`lib/broadlistening/planner.rb`) - Determines which steps need to run based on dependencies
+- **Status** (`lib/broadlistening/status.rb`) - Tracks execution status and locking
+
+### Services
+
+- `LlmClient` - OpenAI API wrapper for chat completions
+- `KMeans` - K-means clustering implementation using Numo::NArray
+- `HierarchicalClustering` - Builds hierarchical cluster structure
+
+### Steps
+
+All steps inherit from `Steps::BaseStep` and implement `execute` method:
+- Steps read from and write to Context
+- Each step has an output file defined in Context::OUTPUT_FILES
+- Dependencies between steps are defined in hierarchical_specs.json
+
+## Technology Stack
+
+- **Ruby**: >= 3.1.0
+- **Numerical Computing**: Numo::NArray
+- **Dimensionality Reduction**: umappp (C++ native extension)
+- **LLM**: ruby-openai
+- **Parallelization**: parallel gem
+- **Schema Validation**: json_schemer
+- **Code Style**: rubocop-rails-omakase, rubocop-rspec
+
+## Kouchou-AI Compatibility
+
+This gem produces output compatible with Kouchou-AI's hierarchical_result.json format:
+- Schema defined in `schema/hierarchical_result.json`
+- Use `Compatibility.validate_with_schema(output)` to validate
+- Step specs loaded from Python's `server/broadlistening/pipeline/hierarchical_specs.json`
+
+### Output Format
+
+Final result includes: arguments, clusters, comments, propertyMap, translations, overview, config
+
+## Native Extension Notes
+
+The umappp gem requires a C++ compiler:
+```bash
+# macOS
+CXX=clang++ gem install umappp
+
+# Use Rice 4.6.x (compatibility issues with 4.7.x)
+```

data/LICENSE

@@ -0,0 +1,24 @@
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007
+
+Copyright (C) 2025 Masayoshi Takahashi
+Based on Kouchou-AI (https://github.com/digitaldemocracy2030/kouchou-ai)
+by Digital Democracy 2030
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published
+by the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+                            ---
+
+The full text of the GNU Affero General Public License version 3
+is included in the file LICENSE-AGPLv3.txt in this repository.