sec_api 1.0.0

data/MIGRATION.md

@@ -0,0 +1,274 @@
+# Migration Guide: v0.1.0 → v1.0.0
+
+This guide documents breaking changes when upgrading from `sec_api` v0.1.0 to v1.0.0 and provides migration examples.
+
+## Overview of Breaking Changes
+
+v1.0.0 standardizes all API responses to return **strongly-typed, immutable objects** instead of raw hashes. This ensures thread safety, improves IDE support, and provides a consistent API surface.
+
+**Affected endpoints:**
+- Mapping endpoints (ticker, cik, cusip, name)
+- Extractor endpoint (extract)
+- FulltextResults collection (now Enumerable)
+
+## Breaking Change #1: Mapping Methods Return Entity Objects
+
+### What Changed
+
+All mapping methods now return `SecApi::Entity` objects instead of raw hashes.
+
+**Affected methods:**
+- `client.mapping.ticker(symbol)`
+- `client.mapping.cik(cik_number)`
+- `client.mapping.cusip(cusip_id)`
+- `client.mapping.name(company_name)`
+
+### Migration Example
+
+**v0.1.0 - Returns Hash:**
+```ruby
+client = SecApi::Client.new(api_key: "your_key")
+entity_data = client.mapping.ticker("AAPL")
+
+# Hash access with string keys
+cik = entity_data["cik"]           # => "0000320193"
+ticker = entity_data["ticker"]     # => "AAPL"
+name = entity_data["name"]         # => "Apple Inc."
+```
+
+**v1.0.0 - Returns Entity Object:**
+```ruby
+client = SecApi::Client.new(api_key: "your_key")
+entity = client.mapping.ticker("AAPL")
+
+# Method access (typed attributes)
+cik = entity.cik           # => "0000320193"
+ticker = entity.ticker     # => "AAPL"
+name = entity.name         # => "Apple Inc."
+
+# Hash access NO LONGER WORKS
+entity["cik"]  # => NoMethodError: undefined method `[]' for SecApi::Entity
+```
+
+### Migration Steps
+
+1. **Replace hash bracket notation with method calls:**
+   ```ruby
+   # BEFORE
+   entity_data["cik"]
+
+   # AFTER
+   entity.cik
+   ```
+
+2. **Update type checks if any:**
+   ```ruby
+   # BEFORE
+   if entity_data.is_a?(Hash)
+
+   # AFTER
+   if entity.is_a?(SecApi::Entity)
+   ```
+
+3. **Immutability note:** Entity objects are frozen and cannot be modified:
+   ```ruby
+   entity.cik = "new_value"  # => FrozenError: can't modify frozen SecApi::Entity
+   ```
+
+## Breaking Change #2: Extractor Returns ExtractedData Objects
+
+### What Changed
+
+The `extract` method now returns `SecApi::ExtractedData` objects instead of raw hashes.
+
+**Affected methods:**
+- `client.extractor.extract(filing_url)`
+
+### Migration Example
+
+**v0.1.0 - Returns Hash:**
+```ruby
+client = SecApi::Client.new(api_key: "your_key")
+filing_url = "https://www.sec.gov/Archives/edgar/data/320193/..."
+
+extracted = client.extractor.extract(filing_url)
+
+# Hash access
+text = extracted["text"]
+sections = extracted["sections"]
+metadata = extracted["metadata"]
+```
+
+**v1.0.0 - Returns ExtractedData Object:**
+```ruby
+client = SecApi::Client.new(api_key: "your_key")
+filing_url = "https://www.sec.gov/Archives/edgar/data/320193/..."
+
+extracted = client.extractor.extract(filing_url)
+
+# Method access (typed attributes)
+text = extracted.text              # => "Full extracted text..."
+sections = extracted.sections      # => { risk_factors: "...", financials: "..." }
+metadata = extracted.metadata      # => { source_url: "...", form_type: "10-K" }
+
+# Hash access NO LONGER WORKS
+extracted["text"]  # => NoMethodError: undefined method `[]' for SecApi::ExtractedData
+```
+
+### Migration Steps
+
+1. **Replace hash bracket notation with method calls:**
+   ```ruby
+   # BEFORE
+   extracted["text"]
+   extracted["sections"]["risk_factors"]
+
+   # AFTER
+   extracted.text
+   extracted.sections[:risk_factors]  # Note: sections keys are symbols in v1.0.0
+   ```
+
+2. **Update type checks:**
+   ```ruby
+   # BEFORE
+   if extracted.is_a?(Hash)
+
+   # AFTER
+   if extracted.is_a?(SecApi::ExtractedData)
+   ```
+
+3. **Handle optional attributes:**
+   ```ruby
+   # All attributes are optional (may be nil)
+   if extracted.text
+     process_text(extracted.text)
+   end
+
+   if extracted.sections
+     risk_factors = extracted.sections[:risk_factors]
+   end
+   ```
+
+## Breaking Change #3: FulltextResults is Enumerable
+
+### What Changed
+
+`SecApi::Collections::FulltextResults` now includes `Enumerable`, allowing direct iteration without calling `.fulltext_results` first.
+
+### Migration Example
+
+**v0.1.0 - Not Enumerable:**
+```ruby
+results = client.query.fulltext("merger acquisition")
+
+# Must use .fulltext_results accessor
+results.fulltext_results.each { |r| puts r.ticker }
+results.fulltext_results.map(&:ticker)
+results.fulltext_results.select { |r| r.form_type == "8-K" }
+```
+
+**v1.0.0 - Enumerable:**
+```ruby
+results = client.query.fulltext("merger acquisition")
+
+# Direct iteration (Enumerable)
+results.each { |r| puts r.ticker }
+results.map(&:ticker)
+results.select { |r| r.form_type == "8-K" }
+
+# .fulltext_results accessor still works for backward compatibility
+results.fulltext_results.each { |r| puts r.ticker }
+```
+
+### Migration Steps
+
+1. **Simplify iteration (optional):**
+   ```ruby
+   # BEFORE (still works)
+   results.fulltext_results.each { |r| ... }
+
+   # AFTER (cleaner)
+   results.each { |r| ... }
+   ```
+
+2. **Use Enumerable methods directly:**
+   ```ruby
+   # BEFORE
+   results.fulltext_results.count
+   results.fulltext_results.first(10)
+
+   # AFTER
+   results.count
+   results.first(10)
+   ```
+
+## Thread Safety Improvements
+
+All response objects in v1.0.0 are **immutable and thread-safe**:
+
+```ruby
+# Safe for concurrent usage (Sidekiq, background jobs, etc.)
+entity = client.mapping.ticker("AAPL")
+
+threads = 10.times.map do
+  Thread.new do
+    100.times { puts entity.cik }
+  end
+end
+
+threads.each(&:join)  # No race conditions
+```
+
+## IDE Autocomplete Benefits
+
+With typed objects, your IDE can now provide accurate autocomplete:
+
+```ruby
+entity = client.mapping.ticker("AAPL")
+entity.  # IDE shows: cik, ticker, name, exchange, sic, etc.
+
+extracted = client.extractor.extract(filing_url)
+extracted.  # IDE shows: text, sections, metadata
+```
+
+## Quick Reference: API Changes
+
+| Endpoint | v0.1.0 Return Type | v1.0.0 Return Type | Migration |
+|----------|-------------------|-------------------|-----------|
+| `mapping.ticker()` | `Hash` | `Entity` | Use `.cik` instead of `["cik"]` |
+| `mapping.cik()` | `Hash` | `Entity` | Use `.ticker` instead of `["ticker"]` |
+| `mapping.cusip()` | `Hash` | `Entity` | Use `.name` instead of `["name"]` |
+| `mapping.name()` | `Hash` | `Entity` | Use `.cik` instead of `["cik"]` |
+| `extractor.extract()` | `Hash` | `ExtractedData` | Use `.text` instead of `["text"]` |
+| `query.fulltext()` | `FulltextResults` (not Enumerable) | `FulltextResults` (Enumerable) | Use `.each` directly |
+
+## Testing Your Migration
+
+After migrating, verify your code works:
+
+```ruby
+# Test mapping endpoints
+entity = client.mapping.ticker("AAPL")
+raise unless entity.is_a?(SecApi::Entity)
+raise unless entity.cik == "0000320193"
+
+# Test extractor endpoint
+extracted = client.extractor.extract(filing_url)
+raise unless extracted.is_a?(SecApi::ExtractedData)
+raise unless extracted.text.is_a?(String) || extracted.text.nil?
+
+# Test collections
+results = client.query.fulltext("acquisition")
+raise unless results.respond_to?(:each)
+raise unless results.first.is_a?(SecApi::FulltextResult)
+```
+
+## Need Help?
+
+- **GitHub Issues:** https://github.com/your-org/sec_api/issues
+- **Documentation:** See README.md and inline YARD docs
+- **Breaking Changes:** This migration guide documents all breaking changes
+
+## Summary
+
+v1.0.0 brings production-grade thread safety and a consistent, typed API surface. While hash access is no longer supported, the migration is straightforward: replace `["key"]` with `.key` for all mapping and extractor responses.

data/README.md

@@ -0,0 +1,370 @@
+# sec_api
+
+[![Gem Version](https://badge.fury.io/rb/sec_api.svg)](https://badge.fury.io/rb/sec_api)
+[![CI](https://github.com/ljuti/sec_api/actions/workflows/main.yml/badge.svg)](https://github.com/ljuti/sec_api/actions/workflows/main.yml)
+[![Ruby](https://img.shields.io/badge/ruby-3.1%2B-ruby.svg)](https://www.ruby-lang.org)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.txt)
+[![Documentation](https://img.shields.io/badge/docs-YARD-blue.svg)](https://rubydoc.info/gems/sec_api)
+
+Production-grade Ruby client for accessing SEC EDGAR filings through the [sec-api.io](https://sec-api.io) API. Query, search, and extract structured financial data from 18+ million SEC filings with automatic retry, rate limiting, and comprehensive error handling.
+
+## Features
+
+- **Query Builder DSL** - Fluent, chainable interface for searching filings by ticker, CIK, form type, date range, and full-text keywords
+- **Automatic Pagination** - Memory-efficient lazy enumeration through large result sets with `auto_paginate`
+- **Entity Mapping** - Resolve tickers, CIKs, CUSIPs, and company names to entity records
+- **XBRL Extraction** - Extract structured financial data from US GAAP and IFRS filings
+- **Real-Time Streaming** - WebSocket notifications for new filings with <2 minute latency
+- **Intelligent Rate Limiting** - Proactive throttling and request queueing to maximize throughput
+- **Production Error Handling** - TransientError/PermanentError hierarchy with automatic retry
+- **Observability Hooks** - Instrumentation callbacks for logging, metrics, and distributed tracing
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem 'sec_api'
+```
+
+Or install directly:
+
+```bash
+gem install sec_api
+```
+
+## Quick Start
+
+### Configuration
+
+Set your API key via environment variable:
+
+```bash
+export SECAPI_API_KEY=your_api_key_here
+```
+
+Get your API key from [sec-api.io](https://sec-api.io).
+
+Alternatively, create `config/secapi.yml`:
+
+```yaml
+api_key: <%= ENV['SECAPI_API_KEY'] %>
+```
+
+### Basic Usage
+
+```ruby
+require 'sec_api'
+
+# Initialize client (auto-loads configuration)
+client = SecApi::Client.new
+
+# Query filings by ticker
+filings = client.query.ticker("AAPL").form_type("10-K").search
+filings.each do |filing|
+  puts "#{filing.form_type} filed on #{filing.filed_at}"
+end
+```
+
+## Usage Examples
+
+### Query Builder
+
+```ruby
+# Simple ticker query
+filings = client.query
+  .ticker("AAPL")
+  .form_type("10-K")
+  .search
+
+# Multiple tickers and form types with date range
+filings = client.query
+  .ticker("AAPL", "TSLA", "GOOGL")
+  .form_type("10-K", "10-Q", "8-K")
+  .date_range(from: "2020-01-01", to: Date.today)
+  .limit(100)
+  .search
+
+# Full-text search
+filings = client.query
+  .ticker("META")
+  .search_text("artificial intelligence")
+  .search
+```
+
+### Automatic Pagination
+
+```ruby
+# Manual pagination
+filings = client.query.ticker("AAPL").search
+next_page = filings.fetch_next_page if filings.has_more?
+
+# Automatic pagination for backfills (memory-efficient)
+client.query
+  .ticker("AAPL")
+  .date_range(from: "2015-01-01", to: Date.today)
+  .auto_paginate
+  .each do |filing|
+    # Process thousands of filings with constant memory usage
+    process_filing(filing)
+  end
+```
+
+### Entity Mapping
+
+```ruby
+# Ticker to entity
+entity = client.mapping.ticker("AAPL")
+puts "CIK: #{entity.cik}, Name: #{entity.name}"
+
+# CIK to entity
+entity = client.mapping.cik("0000320193")
+
+# CUSIP lookup
+entity = client.mapping.cusip("037833100")
+```
+
+### XBRL Data Extraction
+
+```ruby
+# Extract XBRL data from a filing
+xbrl_data = client.xbrl.to_json(filing.xbrl_url)
+
+# Access financial data by US GAAP element names
+revenue = xbrl_data.statements_of_income["RevenueFromContractWithCustomerExcludingAssessedTax"]
+assets = xbrl_data.balance_sheets["Assets"]
+
+# Discover available elements
+xbrl_data.element_names  # => ["Assets", "Revenue", ...]
+xbrl_data.taxonomy_hint  # => :us_gaap or :ifrs
+```
+
+### Real-Time Streaming
+
+```ruby
+# Subscribe to filtered filings
+client.stream.subscribe(
+  tickers: ["AAPL", "TSLA"],
+  form_types: ["10-K", "8-K"]
+) do |filing|
+  puts "New filing: #{filing.ticker} - #{filing.form_type}"
+  ProcessFilingJob.perform_async(filing.accession_no)
+end
+```
+
+### Error Handling
+
+```ruby
+begin
+  filings = client.query.ticker("AAPL").search
+rescue SecApi::RateLimitError => e
+  # Automatically retried with exponential backoff
+  # Only raised after max retries exhausted
+  puts "Rate limited: retry after #{e.retry_after}s"
+rescue SecApi::AuthenticationError => e
+  # Permanent error - fix API key
+  puts "Auth failed: #{e.message}"
+rescue SecApi::TransientError => e
+  # Network or server error - safe to retry
+  retry
+rescue SecApi::PermanentError => e
+  # Don't retry - fix the request
+  puts "Permanent error: #{e.message}"
+end
+```
+
+### Observability
+
+```ruby
+# Configure instrumentation callbacks
+config = SecApi::Config.new(
+  api_key: ENV.fetch("SECAPI_API_KEY"),
+
+  on_request: ->(request_id:, method:, url:, headers:) {
+    Rails.logger.info("SEC API Request", request_id: request_id, url: url)
+  },
+
+  on_response: ->(request_id:, status:, duration_ms:, url:, method:) {
+    StatsD.histogram("sec_api.request.duration_ms", duration_ms)
+  },
+
+  on_error: ->(request_id:, error:, url:, method:) {
+    Bugsnag.notify(error)
+  }
+)
+
+client = SecApi::Client.new(config)
+
+# Or use automatic structured logging
+client = SecApi::Client.new(
+  api_key: ENV.fetch("SECAPI_API_KEY"),
+  logger: Rails.logger,
+  default_logging: true
+)
+```
+
+## Architecture
+
+### Client Proxy Pattern
+
+```ruby
+SecApi::Client
+├── .query      # Query API proxy (fluent search builder)
+├── .mapping    # Mapping API proxy (ticker/CIK resolution)
+├── .extractor  # Extractor API proxy (document extraction)
+├── .xbrl       # XBRL API proxy (financial data)
+└── .stream     # Stream API proxy (WebSocket notifications)
+```
+
+### Exception Hierarchy
+
+```
+SecApi::Error (base)
+├── TransientError (automatic retry)
+│   ├── RateLimitError (429)
+│   ├── ServerError (5xx)
+│   └── NetworkError
+└── PermanentError (fail immediately)
+    ├── AuthenticationError (401/403)
+    ├── NotFoundError (404)
+    ├── ValidationError (400/422)
+    └── ConfigurationError
+```
+
+### Middleware Stack
+
+```
+Request → Instrumentation → Retry → RateLimiter → ErrorHandler → Adapter → sec-api.io
+```
+
+## Configuration Options
+
+All options can be set via YAML or environment variables:
+
+| Option | Env Variable | Default | Description |
+|--------|--------------|---------|-------------|
+| `api_key` | `SECAPI_API_KEY` | _(required)_ | Your sec-api.io API key |
+| `base_url` | `SECAPI_BASE_URL` | `https://api.sec-api.io` | API base URL |
+| `retry_max_attempts` | `SECAPI_RETRY_MAX_ATTEMPTS` | `5` | Maximum retry attempts |
+| `retry_initial_delay` | `SECAPI_RETRY_INITIAL_DELAY` | `1.0` | Initial retry delay (seconds) |
+| `retry_max_delay` | `SECAPI_RETRY_MAX_DELAY` | `60` | Maximum retry delay (seconds) |
+| `request_timeout` | `SECAPI_REQUEST_TIMEOUT` | `30` | HTTP request timeout (seconds) |
+| `rate_limit_threshold` | `SECAPI_RATE_LIMIT_THRESHOLD` | `0.1` | Throttle when <10% quota remains |
+| `default_logging` | - | `false` | Enable automatic structured logging |
+| `metrics_backend` | - | `nil` | StatsD-compatible metrics backend |
+
+## Requirements
+
+- **Ruby:** 3.1.0 or higher
+- **Dependencies:**
+  - `faraday` - HTTP client
+  - `faraday-retry` - Automatic retry middleware
+  - `anyway_config` - Configuration management
+  - `dry-struct` - Immutable value objects
+  - `faye-websocket` - WebSocket client for streaming
+  - `eventmachine` - Event-driven I/O
+
+## Documentation
+
+### API Reference
+
+Generate YARD documentation:
+
+```bash
+bundle exec yard doc
+open doc/index.html
+```
+
+### Usage Examples
+
+See working examples in `docs/examples/`:
+
+| File | Description |
+|------|-------------|
+| [query_builder.rb](docs/examples/query_builder.rb) | Query by ticker, CIK, form type, date range |
+| [backfill_filings.rb](docs/examples/backfill_filings.rb) | Multi-year backfill with auto-pagination |
+| [streaming_notifications.rb](docs/examples/streaming_notifications.rb) | Real-time WebSocket notifications |
+| [instrumentation.rb](docs/examples/instrumentation.rb) | Logging, metrics, and observability |
+
+### Migration Guide
+
+Upgrading from v0.1.0? See the [Migration Guide](docs/migration-guide-v1.md) for breaking changes and upgrade instructions.
+
+### Architecture Documentation
+
+- [Product Requirements](_bmad-output/planning-artifacts/prd.md) - Complete requirements
+- [Architecture](_bmad-output/planning-artifacts/architecture.md) - Technical decisions
+- [Epics & Stories](_bmad-output/planning-artifacts/epics.md) - Implementation roadmap
+
+## Development
+
+### Setup
+
+```bash
+git clone https://github.com/ljuti/sec_api.git
+cd sec_api
+bin/setup
+```
+
+### Testing
+
+```bash
+bundle exec rspec              # Run tests
+bundle exec standardrb         # Run linter
+bundle exec rake               # Run both
+```
+
+### Interactive Console
+
+```bash
+bin/console
+```
+
+## Roadmap
+
+### v0.1.0
+- ✅ Basic query, search, mapping, extractor endpoints
+- ✅ Configuration via anyway_config
+- ✅ Immutable value objects (Dry::Struct)
+
+### v1.0.0
+- ✅ Production-grade error handling with TransientError/PermanentError
+- ✅ Fluent query builder DSL
+- ✅ Automatic pagination with lazy enumeration
+- ✅ XBRL extraction with taxonomy detection
+- ✅ Real-time streaming API (WebSocket)
+- ✅ Intelligent rate limiting with proactive throttling
+- ✅ Observability hooks (instrumentation callbacks)
+- ✅ Structured logging and metrics integration
+- ✅ 100% YARD documentation coverage
+- ✅ Migration guide from v0.1.0
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/ljuti/sec_api.
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/my-feature`)
+3. Write tests for your changes
+4. Ensure tests pass (`bundle exec rspec && bundle exec standardrb`)
+5. Commit your changes (`git commit -am 'Add new feature'`)
+6. Push to the branch (`git push origin feature/my-feature`)
+7. Create a Pull Request
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](LICENSE.txt).
+
+## Support
+
+- **GitHub Issues:** https://github.com/ljuti/sec_api/issues
+- **Author:** Lauri Jutila
+- **Email:** git@laurijutila.com
+
+## Acknowledgments
+
+This gem interacts with the [sec-api.io](https://sec-api.io) API. You'll need an API key from sec-api.io to use this gem.
+
+---
+
+**Status:** v1.0.0 released

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "standard/rake"
+
+task default: %i[spec standard]

data/config/secapi.yml.example

@@ -0,0 +1,57 @@
+# SecAPI Configuration Example
+# Copy this file to config/secapi.yml or config/secapi.local.yml
+# and configure your API credentials
+
+# REQUIRED: Your SEC API key
+# Get your API key from https://sec-api.io
+# Can also be set via SECAPI_API_KEY environment variable
+api_key: <%= ENV.fetch('SECAPI_API_KEY', 'your_api_key_here') %>
+
+# OPTIONAL: Base URL for SEC API endpoints
+# Default: https://api.sec-api.io
+# Override for testing environments or sandbox APIs
+# Can also be set via SECAPI_BASE_URL environment variable
+# base_url: https://api.sec-api.io
+
+# OPTIONAL: Retry Configuration
+# Maximum number of retry attempts for failed requests
+# Default: 5
+# retry_max_attempts: 5
+
+# Initial delay in seconds before first retry
+# Default: 1.0
+# retry_initial_delay: 1.0
+
+# Maximum delay in seconds between retries (exponential backoff cap)
+# Default: 60.0
+# retry_max_delay: 60.0
+
+# OPTIONAL: Timeout Configuration
+# Request timeout in seconds
+# Default: 30
+# request_timeout: 30
+
+# OPTIONAL: Rate Limiting Configuration
+# Threshold (0.0-1.0) for proactive throttling
+# Default: 0.1 (throttle when less than 10% quota remaining)
+# rate_limit_threshold: 0.1
+
+# PRODUCTION DEPLOYMENT EXAMPLE
+# Using environment variables (recommended for production):
+#
+# api_key: <%= ENV.fetch('SECAPI_API_KEY') %>
+# base_url: <%= ENV.fetch('SECAPI_BASE_URL', 'https://api.sec-api.io') %>
+# retry_max_attempts: <%= ENV.fetch('SECAPI_RETRY_MAX_ATTEMPTS', 5).to_i %>
+# request_timeout: <%= ENV.fetch('SECAPI_REQUEST_TIMEOUT', 30).to_i %>
+
+# TESTING ENVIRONMENT EXAMPLE
+# Override base_url to point to test server:
+#
+# base_url: https://test.sec-api.example.com
+# api_key: test_api_key_12345
+
+# LOCAL DEVELOPMENT
+# For local overrides without committing credentials:
+# 1. Copy this file to config/secapi.local.yml
+# 2. Add config/secapi.local.yml to .gitignore (already done)
+# 3. Configure your local API key in secapi.local.yml