sentiment_insights 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '008d6ea6d7343f377abedc02ac6c28bd072c21f8f69d150fc5c5a2dd744fbf96'
-  data.tar.gz: 1cd6db74422570ea2ec02a6e9b8976cd9b8a91120d271ee900057431a4b4665e
+  metadata.gz: 74729added1e999a0e6277b89a95fb2ee934103f18df50b8187f6a86d04c833b
+  data.tar.gz: 3be0d99ae0ecc2148110d64bf7030d988e9a03810774d338e1c1ac58d735ed48
 SHA512:
-  metadata.gz: e393148073465c4f022bff7440d11df6e2249b77aee022a2f7c6a6d941a7271c1541b331688f00c5d9cdf3928fcfb45dec8a087ea151316595830b769c9974e1
-  data.tar.gz: b8229c858ff529e53144214d567d679eb608c3f03dc5779a3541f07a2167e3eba6c56090bc0260b1c92ca7a32e4b2572ecf8ce5db315802c2a44b7f878c04e2c
+  metadata.gz: 563aecf46e7af1300bb84854f8af8c30eeaa874bf73eb49f18c44c5b2af075084b4fe8eaaf1503b43dad87e48225601f358c99caf658abdad5fd30423d799e22
+  data.tar.gz: 5f017d18b1a447e22399d161f493e3f01784eb917790b846548c0d63aef13f0b8df58101e197c0625c4dbfc209e34e5072a533f41026238468ad6e9fa1bb90ad

data/EXPORT_USAGE.md

@@ -0,0 +1,325 @@
+# SentimentInsights Export Feature Guide
+
+The SentimentInsights gem now includes comprehensive export functionality for all analysis results. Export your sentiment analysis, entity extraction, and key phrase results to CSV and Excel formats with advanced filtering and customization options.
+
+## 🚀 Quick Start
+
+```ruby
+# Run your analysis as usual
+result = SentimentInsights::Insights::Sentiment.new.analyze(entries)
+
+# Export to CSV (simplest usage)
+result.to_csv("my_analysis.csv")
+
+# Export to Excel
+result.to_excel("my_analysis.xlsx")
+
+# Auto-detect format from filename
+result.export("my_analysis.csv")  # Creates CSV
+result.export("my_analysis.xlsx") # Creates Excel
+```
+
+## 📊 CSV Export Structure
+
+### Sentiment Analysis CSV
+```csv
+response_id,text,sentiment_label,sentiment_score,segment_age_group,segment_region,timestamp
+r_1,"I love this product!",positive,0.9,18-25,North,2024-06-28T10:30:00Z
+r_2,"Service was terrible",negative,-0.8,26-35,South,2024-06-28T10:31:00Z
+
+SUMMARY STATISTICS
+Total Responses,150
+Positive Count,90
+Positive Percentage,60.0%
+Net Sentiment Score,40.0
+
+SEGMENT ANALYSIS  
+Segment Type,Segment Value,Total Count,Positive %,Neutral %,Negative %,Net Score
+Age Group,18-25,75,65.3%,18.7%,16.0%,49.3
+Age Group,26-35,45,53.3%,22.2%,24.4%,28.9
+```
+
+### Entity Extraction CSV
+```csv
+response_id,text,entities_found,segment_age_group,segment_region,timestamp
+r_1,"Apple released iPhone",apple:ORGANIZATION,iphone:PRODUCT,18-25,North,2024-06-28T10:30:00Z
+
+SUMMARY STATISTICS
+Total Unique Entities,25
+Organizations Mentioned,8
+Most Mentioned Entity,apple (5 times)
+
+ENTITY DETAILS
+Entity,Type,Total Mentions,Response IDs,Segment Distribution
+apple,ORGANIZATION,5,"r_1,r_3,r_7",age_group_18-25:3,age_group_26-35:2
+```
+
+### Key Phrases CSV
+```csv
+response_id,text,sentiment,sentiment_score,key_phrases_found,segment_age_group,timestamp
+r_1,"Great customer service!",positive,0.8,"customer service,great experience",18-25,2024-06-28T10:30:00Z
+
+PHRASE DETAILS
+Phrase,Total Mentions,Response IDs,Sentiment Distribution,Segment Distribution
+customer service,8,"r_1,r_4,r_7",positive:5,neutral:2,negative:1,age_group_18-25:4
+```
+
+## 🎛️ Advanced Export Options
+
+### Basic Configuration
+```ruby
+# Export with custom options
+result.to_csv("analysis.csv", {
+  include_summary: true,      # Include summary statistics
+  include_segments: true,     # Include segment analysis  
+  include_timestamp: true,    # Add timestamp column
+  timestamp_format: "%Y-%m-%d %H:%M:%S"
+})
+
+# Skip summary and segments (responses only)
+result.to_csv("responses_only.csv", {
+  include_summary: false,
+  include_segments: false
+})
+```
+
+### Dynamic Segment Handling
+The export system automatically detects and flattens all segment structures:
+
+```ruby
+entries = [
+  { 
+    answer: "Great product!", 
+    segment: { 
+      demographics: { age: "25-34", income: "high" },
+      location: { country: "USA", state: "CA" },
+      simple_field: "value"
+    }
+  }
+]
+
+result = analyzer.analyze(entries)
+result.to_csv("complex_segments.csv")
+
+# CSV headers will be:
+# response_id,text,sentiment_label,sentiment_score,
+# segment_demographics_age,segment_demographics_income,
+# segment_location_country,segment_location_state,
+# segment_simple_field,timestamp
+```
+
+## 🔍 Filtering and Custom Exports
+
+### Fluent Interface
+```ruby
+# Method chaining for custom exports
+result
+  .responses_only                    # Skip summary sections
+  .filter_sentiment(:positive)       # Only positive responses
+  .filter_segments(age_group: ["18-25", "26-35"])  # Specific age groups
+  .to_csv("young_positive_feedback.csv")
+```
+
+### Quick Filter Methods
+```ruby
+# Export only positive responses
+result.export_positive(:csv, "positive_feedback.csv")
+
+# Export only negative responses  
+result.export_negative(:excel, "issues_to_address.xlsx")
+
+# Export specific segments
+result.export_by_segment(:age_group, ["18-25"], :csv, "young_users.csv")
+result.export_by_segment(:region, ["North", "South"], :excel, "regional_analysis.xlsx")
+```
+
+### Advanced Filtering
+```ruby
+# Complex filter combinations
+result.exporter({
+  filter: {
+    sentiment: [:positive, :neutral],  # Exclude negative
+    segments: { 
+      age_group: ["18-25", "26-35"],
+      region: ["North", "West"] 
+    }
+  },
+  include_summary: false
+}).to_csv("filtered_analysis.csv")
+```
+
+## 📈 Excel Export Features
+
+Excel exports include multiple worksheets:
+
+1. **Responses** - Individual response data
+2. **Summary** - Statistical summaries  
+3. **Segment Analysis** - Breakdown by segments
+4. **Entity Details** - Entity-specific analysis (entities only)
+5. **Phrase Details** - Phrase-specific analysis (key phrases only)
+
+```ruby
+# Basic Excel export
+result.to_excel("comprehensive_report.xlsx")
+
+# Excel with custom options
+result.to_excel("detailed_report.xlsx", {
+  include_summary: true,
+  include_segments: true
+})
+```
+
+## 🚀 Batch Export
+
+Export to multiple formats at once:
+
+```ruby
+# Export to both CSV and Excel
+files = result.export_all("quarterly_analysis")
+# Returns: { csv: "quarterly_analysis.csv", excel: "quarterly_analysis.xlsx" }
+
+files.each do |format, filename|
+  puts "#{format.upcase} exported to: #{filename}"
+end
+```
+
+## 🔧 Advanced Usage with Exporter Class
+
+For maximum control, use the Exporter class directly:
+
+```ruby
+exporter = result.exporter({
+  include_summary: true,
+  include_segments: true,
+  filter: { sentiment: [:positive] }
+})
+
+# Multiple export operations
+csv_file = exporter.to_csv("positive_analysis.csv")
+excel_file = exporter.to_excel("positive_analysis.xlsx")
+
+# Specialized exports
+summary_only = exporter.to_csv_summary_only("summary.csv")
+responses_only = exporter.to_csv_responses_only("responses.csv")
+```
+
+## 🎯 Use Cases
+
+### Customer Feedback Analysis
+```ruby
+feedback = analyzer.analyze(customer_responses)
+
+# Export all feedback
+feedback.export_all("customer_feedback_analysis")
+
+# Export issues that need attention
+feedback.export_negative(:excel, "issues_to_resolve.xlsx")
+
+# Export by customer segment
+feedback.export_by_segment(:customer_tier, ["premium", "enterprise"], :csv, "premium_feedback.csv")
+```
+
+### Survey Response Processing
+```ruby
+survey_results = analyzer.analyze(survey_responses)
+
+# Comprehensive report for stakeholders
+survey_results.to_excel("survey_report.xlsx", {
+  include_summary: true,
+  include_segments: true
+})
+
+# Quick CSV for data team
+survey_results
+  .responses_only
+  .to_csv("raw_survey_data.csv")
+```
+
+### Regional Analysis
+```ruby
+regional_analysis = analyzer.analyze(regional_data)
+
+# Export by region
+%w[North South East West].each do |region|
+  regional_analysis.export_by_segment(:region, region, :csv, "#{region.downcase}_analysis.csv")
+end
+```
+
+## 📋 File Naming
+
+Auto-generated filenames include timestamps:
+- `sentiment_analysis_20240628_103045.csv`
+- `entities_analysis_20240628_103045.xlsx`
+- `key_phrases_analysis_20240628_103045.csv`
+
+## 🛠️ Installation Requirements
+
+Add to your Gemfile:
+```ruby
+gem 'sentiment_insights', '~> 0.3.0'
+gem 'rubyXL', '~> 3.4'  # For Excel export support
+```
+
+## 🔍 Error Handling
+
+```ruby
+begin
+  result.to_excel("report.xlsx")
+rescue => e
+  puts "Excel export failed: #{e.message}"
+  # Fallback to CSV
+  result.to_csv("report.csv")
+end
+```
+
+## 🎉 Integration Examples
+
+### Rails Controller
+```ruby
+class AnalyticsController < ApplicationController
+  def export_feedback
+    entries = FeedbackResponse.includes(:customer).map do |response|
+      {
+        answer: response.comment,
+        segment: {
+          customer_tier: response.customer.tier,
+          region: response.customer.region,
+          product: response.product_name
+        }
+      }
+    end
+
+    result = SentimentInsights::Insights::Sentiment.new.analyze(entries)
+    
+    filename = result.to_excel("feedback_analysis_#{Date.current}.xlsx")
+    
+    send_file filename, 
+      filename: "customer_feedback_analysis.xlsx",
+      type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
+  end
+end
+```
+
+### Background Job
+```ruby
+class AnalysisExportJob < ApplicationJob
+  def perform(survey_id, email)
+    survey = Survey.find(survey_id)
+    entries = survey.responses.map { |r| { answer: r.text, segment: r.metadata } }
+    
+    result = SentimentInsights::Insights::Sentiment.new.analyze(entries)
+    files = result.export_all("survey_#{survey_id}_analysis")
+    
+    AnalysisMailer.send_results(email, files).deliver_now
+  end
+end
+```
+
+## 🚀 Performance Tips
+
+1. **Use filtering** to reduce export size for large datasets
+2. **Skip unnecessary sections** with `responses_only` for faster exports
+3. **Batch processing** - split large datasets before analysis
+4. **Background jobs** for large exports in web applications
+
+The export system is designed to handle dynamic segment structures, large datasets, and provides maximum flexibility for different use cases while maintaining excellent performance.

data/Gemfile.lock

@@ -1,8 +1,9 @@
 PATH
   remote: .
   specs:
-    sentiment_insights (0.1.0)
+    sentiment_insights (0.4.0)
       aws-sdk-comprehend (>= 1.98.0)
+      rubyXL (~> 3.4)
       sentimental (~> 1.4.0)
 
 GEM
@@ -27,6 +28,9 @@ GEM
     dotenv (2.8.1)
     jmespath (1.6.2)
     logger (1.7.0)
+    nokogiri (1.15.7-arm64-darwin)
+      racc (~> 1.4)
+    racc (1.8.1)
     rake (13.2.1)
     rspec (3.13.0)
       rspec-core (~> 3.13.0)
@@ -41,6 +45,10 @@ GEM
       diff-lcs (>= 1.2.0, < 2.0)
       rspec-support (~> 3.13.0)
     rspec-support (3.13.2)
+    rubyXL (3.4.33)
+      nokogiri (>= 1.10.8)
+      rubyzip (>= 1.3.0)
+    rubyzip (2.4.1)
     sentimental (1.4.1)
 
 PLATFORMS

data/README.md

@@ -1,6 +1,6 @@
 # SentimentInsights
 
-**SentimentInsights** is a Ruby gem for extracting sentiment, key phrases, and named entities from survey responses or free-form textual data. It offers a plug-and-play interface to different NLP providers, including OpenAI and AWS.
+**SentimentInsights** is a Ruby gem for extracting sentiment, key phrases, and named entities from survey responses or free-form textual data. It offers a plug-and-play interface to different NLP providers, including OpenAI, Claude AI, and AWS.
 
 ---
 
@@ -12,6 +12,7 @@
     - [Sentiment Analysis](#sentiment-analysis)
     - [Key Phrase Extraction](#key-phrase-extraction)
     - [Entity Extraction](#entity-extraction)
+- [Export & Output Formats](#export--output-formats)
 - [Provider Options & Custom Prompts](#provider-options--custom-prompts)
 - [Full Example](#full-example)
 - [Contributing](#contributing)
@@ -43,7 +44,7 @@ gem install sentiment_insights
 
 ## Configuration
 
-Configure the provider and (if using OpenAI or AWS) your API key:
+Configure the provider and (if using OpenAI, Claude AI, or AWS) your API key:
 
 ```ruby
 require 'sentiment_insights'
@@ -54,6 +55,12 @@ SentimentInsights.configure do |config|
   config.openai_api_key = ENV["OPENAI_API_KEY"]
 end
 
+# For Claude AI
+SentimentInsights.configure do |config|
+  config.provider = :claude
+  config.claude_api_key = ENV["CLAUDE_API_KEY"]
+end
+
 # For AWS
 SentimentInsights.configure do |config|
   config.provider = :aws
@@ -68,6 +75,7 @@ end
 
 Supported providers:
 - `:openai`
+- `:claude`
 - `:aws`
 - `:sentimental` (local fallback, limited feature set)
 
@@ -121,11 +129,11 @@ result = insight.analyze(
 ```
 
 #### Available Options (`analyze`)
-| Option        | Type    | Description                                                            | Provider    |
-|---------------|---------|------------------------------------------------------------------------|-------------|
-| `question`    | String  | Contextual question for the batch                                     | OpenAI only |
-| `prompt`      | String  | Custom prompt text for LLM                                            | OpenAI only |
-| `batch_size`  | Integer | Number of entries per OpenAI completion call (default: 50)           | OpenAI only |
+| Option        | Type    | Description                                                            | Provider           |
+|---------------|---------|------------------------------------------------------------------------|--------------------|
+| `question`    | String  | Contextual question for the batch                                     | OpenAI, Claude only |
+| `prompt`      | String  | Custom prompt text for LLM                                            | OpenAI, Claude only |
+| `batch_size`  | Integer | Number of entries per completion call (default: 50)                  | OpenAI, Claude only |
 
 #### 📾 Sample Output
 
@@ -211,11 +219,11 @@ result = insight.extract(
 ```
 
 #### Available Options (`extract`)
-| Option             | Type    | Description                                                | Provider     |
-|--------------------|---------|------------------------------------------------------------|--------------|
-| `question`         | String  | Context question to help guide phrase extraction           | OpenAI only  |
-| `key_phrase_prompt`| String  | Custom prompt for extracting key phrases                   | OpenAI only  |
-| `sentiment_prompt` | String  | Custom prompt for classifying tone of extracted phrases    | OpenAI only  |
+| Option             | Type    | Description                                                | Provider           |
+|--------------------|---------|------------------------------------------------------------|--------------------|
+| `question`         | String  | Context question to help guide phrase extraction           | OpenAI, Claude only |
+| `key_phrase_prompt`| String  | Custom prompt for extracting key phrases                   | OpenAI, Claude only |
+| `sentiment_prompt` | String  | Custom prompt for classifying tone of extracted phrases    | OpenAI, Claude only |
 
 #### 📾 Sample Output
 
@@ -267,10 +275,10 @@ result = insight.extract(
 ```
 
 #### Available Options (`extract`)
-| Option      | Type    | Description                                       | Provider     |
-|-------------|---------|---------------------------------------------------|--------------|
-| `question`  | String  | Context question to guide entity extraction       | OpenAI only  |
-| `prompt`    | String  | Custom instructions for OpenAI entity extraction  | OpenAI only  |
+| Option      | Type    | Description                                       | Provider           |
+|-------------|---------|---------------------------------------------------|--------------------|
+| `question`  | String  | Context question to guide entity extraction       | OpenAI, Claude only |
+| `prompt`    | String  | Custom instructions for entity extraction         | OpenAI, Claude only |
 
 #### 📾 Sample Output
 
@@ -306,11 +314,92 @@ result = insight.extract(
          "the response was copy-paste and didn't address my issue directly.",
      :segment=>{:age=>"45-54", :region=>"Midwest"}}]}
 ```
+
+---
+
+## Export & Output Formats
+
+SentimentInsights provides flexible export options for all analysis results. Export to CSV, Excel, or JSON formats with advanced filtering and customization.
+
+### Quick Export Examples
+
+```ruby
+result = SentimentInsights::Insights::Sentiment.new.analyze(entries)
+
+# Direct format methods
+result.to_csv("analysis.csv")           # CSV file
+result.to_json("analysis.json")         # JSON file  
+result.to_excel("analysis.xlsx")        # Excel file
+
+# Auto-detect format from filename
+result.export("analysis.csv")           # Creates CSV
+result.export("analysis.json")          # Creates JSON
+result.export("analysis.xlsx")          # Creates Excel
+
+# API-friendly formats (no files)
+api_data = result.to_hash               # Structured Hash
+json_string = result.to_json_string     # JSON string
+```
+
+### Advanced Export Options
+
+```ruby
+# Filtered exports
+result.export_positive(:csv, "positive_feedback.csv")
+result.export_negative(:excel, "issues_to_address.xlsx")
+
+# Fluent interface with filtering
+result
+  .filter_sentiment(:positive)
+  .filter_segments(age_group: ["18-25", "26-35"])
+  .to_excel("young_positive_feedback.xlsx")
+
+# Export all formats at once
+files = result.export_all("comprehensive_analysis")
+# Returns: { csv: "file.csv", json: "file.json", excel: "file.xlsx" }
+```
+
+### CSV Export Structure
+
+```csv
+response_id,text,sentiment_label,sentiment_score,segment_age_group,segment_region,timestamp
+r_1,"I love this product!",positive,0.9,18-25,North,2024-06-28T10:30:00Z
+
+SUMMARY STATISTICS
+Total Responses,150
+Positive Percentage,60.0%
+Net Sentiment Score,40.0
+
+SEGMENT ANALYSIS
+Segment Type,Segment Value,Total Count,Positive %,Net Score
+Age Group,18-25,75,65.3%,49.3
+```
+
+### JSON Export Structure
+
+```json
+{
+  "metadata": {
+    "export_timestamp": "2024-06-28T10:30:00Z",
+    "analysis_type": "sentiment",
+    "total_responses": 150,
+    "provider_used": "claude"
+  },
+  "analysis": {
+    "responses": [...],
+    "global_summary": {...},
+    "segment_summary": {...}
+  }
+}
+```
+
+**📋 For complete export documentation, see [EXPORT_USAGE.md](EXPORT_USAGE.md)**
+
 ---
 
 ## Provider Options & Custom Prompts
 
-> ⚠️ All advanced options (`question`, `prompt`, `key_phrase_prompt`, `sentiment_prompt`, `batch_size`) apply only to the `:openai` provider.  
+> ⚠️ All advanced options (`question`, `prompt`, `key_phrase_prompt`, `sentiment_prompt`, `batch_size`) apply only to the `:openai` and `:claude` providers.  
 > They are safely ignored for `:aws` and `:sentimental`.
 
 ---
@@ -323,6 +412,12 @@ result = insight.extract(
 OPENAI_API_KEY=your_openai_key_here
 ```
 
+### Claude AI
+
+```bash
+CLAUDE_API_KEY=your_claude_key_here
+```
+
 ### AWS Comprehend
 
 ```bash
@@ -352,8 +447,8 @@ bundle exec rspec
 - [x] Sentiment Analysis
 - [x] Key Phrase Extraction
 - [x] Entity Recognition
+- [x] Export Functionality (CSV, Excel, JSON)
 - [ ] Topic Modeling
-- [ ] CSV/JSON Export Helpers
 - [ ] Visual Dashboard Add-on
 
 ---
@@ -373,6 +468,7 @@ Pull requests welcome! Please open an issue to discuss major changes first.
 ## 💬 Acknowledgements
 
 - [OpenAI GPT](https://platform.openai.com/docs)
+- [Claude AI](https://docs.anthropic.com/claude/reference/getting-started-with-the-api)
 - [AWS Comprehend](https://docs.aws.amazon.com/comprehend/latest/dg/what-is.html)
 - [Sentimental Gem](https://github.com/7compass/sentimental)