sec_api 1.0.0

data/docs/pre-review-checklist.md

@@ -0,0 +1,145 @@
+# Pre-Review Checklist
+
+Use this checklist before submitting code for review. These items were identified through Epic 3 and Epic 4 retrospectives as common issues caught in code review.
+
+## Thread Safety
+
+- [ ] **Immutable objects are frozen** - All Dry::Struct value objects should be deeply frozen
+- [ ] **String attributes frozen** - Use `attribute&.freeze` for string attributes in value objects
+- [ ] **Shared state protected** - Any state shared across requests uses Mutex or is immutable
+- [ ] **No instance variable mutation** - Value objects don't mutate after construction
+
+```ruby
+# Good: Deep freeze in from_api
+def self.from_api(data)
+  new(
+    statements_of_income: parse_statement(data["StatementsOfIncome"]).freeze,
+    # ...
+  ).freeze
+end
+
+# Good: String attribute frozen
+attribute? :text, Types::String.optional
+# In from_api:
+text: data["text"]&.freeze
+```
+
+## Test Coverage
+
+- [ ] **Unit tests for new value objects** - Every new Dry::Struct class needs dedicated tests
+- [ ] **Edge case tests included**:
+  - [ ] `nil` input handling
+  - [ ] Empty string/hash/array input
+  - [ ] Whitespace-only strings
+  - [ ] Invalid type input
+- [ ] **Error message verification** - Exception tests verify message content, not just exception type
+- [ ] **`stubs.verify_stubbed_calls`** - All HTTP stub tests verify stubs were called
+
+```ruby
+# Good: Verify error message content
+expect { method_call }.to raise_error(SecApi::ValidationError) do |error|
+  expect(error.message).to include("missing required field")
+  expect(error.message).to include("period")
+end
+
+# Good: Verify stubs in after block or inline
+after { stubs.verify_stubbed_calls }
+```
+
+## Input Validation
+
+- [ ] **Validate required parameters** - Raise ArgumentError or ValidationError for missing required input
+- [ ] **URL format validation** - SEC URLs follow predictable patterns
+- [ ] **CIK format handling** - Normalize to 10-digit with leading zeros
+- [ ] **Accession number format** - Support both dashed and undashed formats
+
+```ruby
+# Good: Input validation with helpful error
+def ticker(symbol)
+  raise ArgumentError, "ticker symbol required" if symbol.nil? || symbol.empty?
+  # ...
+end
+```
+
+## Documentation
+
+- [ ] **YARD docs on public methods** - `@param`, `@return`, `@raise`, `@example` tags
+- [ ] **Examples match implementation** - YARD examples actually work with current code
+- [ ] **@note for non-obvious behavior** - Document gotchas, limitations, or surprising behavior
+
+```ruby
+# Good: Complete YARD documentation
+# Extracts XBRL data from a filing
+#
+# @param filing_url [String] URL to the SEC filing
+# @return [XbrlData] Typed XBRL data object
+# @raise [NotFoundError] when filing URL is invalid
+# @raise [ValidationError] when response structure is invalid
+#
+# @example Extract from URL
+#   data = client.xbrl.to_json("https://sec.gov/...")
+#   data.statements_of_income["Revenue"]
+#
+# @note Element names follow US-GAAP taxonomy exactly
+#
+def to_json(filing_url)
+```
+
+## Code Quality
+
+- [ ] **DRY - No duplicate code** - Extract shared logic to modules or helpers
+- [ ] **Consistent patterns** - Follow established `from_api` factory method pattern
+- [ ] **No dead code** - Remove commented-out code, unused variables, unreachable branches
+- [ ] **StandardRB passes** - Run `bundle exec standardrb` before review
+
+## Error Handling
+
+- [ ] **Correct exception types**:
+  - `ValidationError` - Malformed data, failed validation
+  - `NotFoundError` - Resource not found (404), invalid URL
+  - `AuthenticationError` - Invalid API key (401, 403)
+  - `NetworkError` - Connection failures, timeouts
+  - `ServerError` - API server errors (500-504)
+- [ ] **Actionable error messages** - Include what failed, why, and received data context
+- [ ] **No silent failures** - All errors raised or logged, never swallowed
+
+```ruby
+# Good: Actionable error message
+raise ValidationError, "XBRL fact missing required 'period' field. " \
+  "Received: #{data.inspect}"
+```
+
+## HTTP Testing Patterns
+
+- [ ] **Use stub_connection/build_connection helpers** - See `spec/support/test_helpers.rb`
+- [ ] **Include ErrorHandler middleware** - Required for error scenario tests
+- [ ] **JSON content type in responses** - `{"Content-Type" => "application/json"}`
+- [ ] **Response body as JSON string** - Use `.to_json` on response hashes
+
+```ruby
+# Good: Complete stub setup
+stubs = Faraday::Adapter::Test::Stubs.new
+stubs.get("/endpoint") do
+  [200, {"Content-Type" => "application/json"}, {data: "value"}.to_json]
+end
+stub_connection(stubs)
+
+# ... test code ...
+
+stubs.verify_stubbed_calls
+```
+
+## Quick Self-Review
+
+Before requesting review, ask yourself:
+
+1. **Would this break in a multi-threaded environment?** (Sidekiq, concurrent requests)
+2. **What happens if the API returns unexpected data?** (nil, wrong type, missing fields)
+3. **Can someone understand this code without context?** (clear names, comments where needed)
+4. **Did I test the failure paths, not just success?**
+5. **Does the documentation match what the code actually does?**
+
+---
+
+*Checklist created from Epic 3 & 4 retrospective learnings*
+*Last updated: 2026-01-07*

data/docs/project-overview.md

@@ -0,0 +1,90 @@
+# sec_api - Project Overview
+
+**Generated:** 2026-01-05
+**Scan Level:** Quick
+**Project Type:** Ruby Library (Gem)
+
+## Executive Summary
+
+sec_api is a production-grade Ruby client library for accessing SEC EDGAR filings through the sec-api.io API. The project is evolving from a basic v0.1.0 API wrapper into enterprise-level financial data infrastructure for commercial financial analysis platforms.
+
+**Current State:** Active development, transitioning from v0.1.0 to v1.0.0
+**Purpose:** Provide Ruby developers with reliable, resilient access to SEC filing data for financial analysis applications
+
+## Project Classification
+
+| Attribute | Value |
+|-----------|-------|
+| **Repository Type** | Monolith |
+| **Project Type** | Library (Ruby Gem) |
+| **Primary Language** | Ruby 3.2.3 |
+| **Minimum Ruby** | 3.1.0+ |
+| **Distribution** | RubyGems (rubygems.org) |
+| **Architecture Pattern** | Client → Proxy pattern with middleware stack |
+
+## Technology Stack
+
+### Core Dependencies
+- **faraday** - HTTP client with middleware capabilities
+- **faraday-retry** - Automatic retry middleware
+- **anyway_config** - Configuration management (YAML + environment variables)
+- **dry-struct** - Immutable value objects for type-safe responses
+
+### Development Dependencies
+- **rspec** ~> 3.0 - Testing framework
+- **standard** ~> 1.3 - Ruby linter (Standard Ruby style guide)
+- **async-http-faraday** - Async HTTP support for concurrent requests
+
+### Key Features (v1.0.0 Target)
+- ✅ Configuration management with validation
+- ✅ Exception hierarchy (TransientError/PermanentError)
+- 🚧 Query builder DSL with fluent interface
+- 🚧 Automatic pagination for large result sets
+- 🚧 XBRL data extraction with validation
+- 🚧 Real-time filing notifications (WebSocket streaming)
+- 🚧 Rate limiting intelligence with proactive throttling
+- 🚧 Production observability hooks
+
+## Project Structure
+
+```
+sec_api/
+├── lib/sec_api/          # Main library code
+│   ├── client.rb         # Client entry point
+│   ├── config.rb         # Configuration (anyway_config)
+│   ├── errors/           # Exception hierarchy
+│   ├── middleware/       # Faraday middleware (retry, rate limiting)
+│   ├── collections/      # Collection objects (Filings, etc.)
+│   └── objects/          # Value objects (Filing, Entity, etc.)
+├── spec/                 # RSpec tests
+├── config/               # Configuration files
+└── _bmad-output/         # Planning artifacts (PRD, Architecture, Epics)
+```
+
+## Development Status
+
+### Completed (v0.1.0)
+- Basic query, search, mapping, and extractor endpoints
+- Configuration via anyway_config
+- Immutable value objects with Dry::Struct
+
+### In Progress (v1.0.0)
+- Production-grade error handling and retry logic
+- Query builder DSL
+- XBRL extraction with validation
+- Real-time streaming API
+- Rate limiting middleware
+- Observability and instrumentation
+
+## Links to Planning Documents
+
+- **[PRD](../_bmad-output/planning-artifacts/prd.md)** - Complete product requirements
+- **[Architecture](../_bmad-output/planning-artifacts/architecture.md)** - Architectural decisions and patterns
+- **[Epics & Stories](../_bmad-output/planning-artifacts/epics.md)** - Implementation breakdown
+
+## Links to Documentation
+
+- [README](../README.md) - Project overview and usage
+- [CHANGELOG](../CHANGELOG.md) - Version history
+- [Source Tree Analysis](./source-tree-analysis.md) - Code organization
+- [Development Guide](./development-guide.md) - Setup and development workflow

data/docs/project-scan-report.json

@@ -0,0 +1,60 @@
+{
+  "workflow_version": "1.2.0",
+  "timestamps": {
+    "started": "2026-01-05T03:28:45Z",
+    "last_updated": "2026-01-05T03:42:00Z",
+    "completed": "2026-01-05T03:42:00Z"
+  },
+  "mode": "initial_scan",
+  "scan_level": "quick",
+  "project_root": "/Users/ljuti/Code/projects/metalsmoney/ruby/sec_api",
+  "output_folder": "/Users/ljuti/Code/projects/metalsmoney/ruby/sec_api/docs",
+  "completed_steps": [
+    {
+      "step": "step_1",
+      "status": "completed",
+      "timestamp": "2026-01-05T03:36:30Z",
+      "summary": "Classified as monolith with 1 part"
+    },
+    {
+      "step": "step_2",
+      "status": "completed",
+      "timestamp": "2026-01-05T03:38:00Z",
+      "summary": "Found 5 existing docs (README, CHANGELOG, PRD, Architecture, Epics)"
+    },
+    {
+      "step": "step_3",
+      "status": "completed",
+      "timestamp": "2026-01-05T03:40:00Z",
+      "summary": "Tech stack: Ruby 3.2.3, Faraday, anyway_config, dry-struct"
+    },
+    {
+      "step": "step_complete",
+      "status": "completed",
+      "timestamp": "2026-01-05T03:42:00Z",
+      "summary": "Documentation generation complete, 4 files written"
+    }
+  ],
+  "current_step": "completed",
+  "findings": {
+    "project_classification": "Monolith, 1 part, Ruby library",
+    "existing_docs_count": 5,
+    "technology_stack": "Ruby 3.2.3, Faraday, anyway_config, dry-struct, RSpec",
+    "architecture_pattern": "Client → Proxy pattern with Faraday middleware"
+  },
+  "project_types": [
+    {
+      "part_id": "main",
+      "project_type_id": "library",
+      "display_name": "sec_api (Ruby Gem)"
+    }
+  ],
+  "outputs_generated": [
+    "project-scan-report.json",
+    "index.md",
+    "project-overview.md",
+    "source-tree-analysis.md",
+    "development-guide.md"
+  ],
+  "resume_instructions": "Workflow complete"
+}

data/docs/source-tree-analysis.md

@@ -0,0 +1,190 @@
+# sec_api - Source Tree Analysis
+
+**Generated:** 2026-01-05
+**Project Root:** `/Users/ljuti/Code/projects/metalsmoney/ruby/sec_api`
+
+## Directory Structure with Annotations
+
+```
+sec_api/
+├── .github/
+│   └── workflows/           # CI/CD workflows (planned for v1.0)
+├── .git/                    # Git repository
+├── .ruby-lsp/              # Ruby LSP cache
+├── _bmad/                  # BMAD framework installation
+│   ├── core/               # Core BMAD modules
+│   └── bmm/                # BMM (BMAD Methodology Manager) modules
+├── _bmad-output/           # BMAD planning artifacts
+│   ├── planning-artifacts/  # PRD, Architecture, Epics documents
+│   └── implementation-artifacts/  # (future: sprint status, stories)
+├── bin/
+│   ├── console             # Interactive IRB console for development
+│   └── setup               # Setup script for dependencies
+├── config/
+│   └── secapi.yml          # Configuration file (API key, settings)
+├── docs/                   # PROJECT DOCUMENTATION (this directory)
+│   ├── project-overview.md  # Project overview (generated)
+│   ├── source-tree-analysis.md  # This file
+│   └── project-scan-report.json  # Workflow state tracking
+├── lib/                    # MAIN LIBRARY CODE
+│   ├── sec_api.rb          # Main entry point (requires all files)
+│   └── sec_api/            # Library modules
+│       ├── client.rb       # 🔑 Client entry point (delegates to proxies)
+│       ├── config.rb       # Configuration management (anyway_config)
+│       ├── version.rb      # Gem version constant
+│       ├── collections/    # Collection objects (Enumerable wrappers)
+│       │   ├── filings.rb  # Filings collection
+│       │   └── fulltext_results.rb  # Full-text search results
+│       ├── errors/         # 🔑 Exception hierarchy
+│       │   ├── error.rb    # Base SecApi::Error
+│       │   ├── transient_error.rb  # Retryable errors
+│       │   ├── permanent_error.rb  # Non-retryable errors
+│       │   ├── rate_limit_error.rb
+│       │   ├── server_error.rb
+│       │   ├── network_error.rb
+│       │   ├── authentication_error.rb
+│       │   ├── not_found_error.rb
+│       │   ├── validation_error.rb
+│       │   └── configuration_error.rb
+│       ├── middleware/     # 🔑 Faraday middleware stack
+│       │   └── error_handler.rb  # HTTP status → exception mapping
+│       ├── objects/        # Value objects (Dry::Struct)
+│       │   ├── filing.rb   # Filing metadata
+│       │   ├── entity.rb   # Company/entity information
+│       │   ├── fulltext_result.rb
+│       │   ├── data_file.rb
+│       │   └── document_format_file.rb
+│       ├── query.rb        # Query API proxy
+│       ├── mapping.rb      # Mapping API proxy (ticker/CIK resolution)
+│       ├── extractor.rb    # Extractor API proxy
+│       └── xbrl.rb         # XBRL API proxy
+├── sig/                    # RBS type signatures (optional)
+├── spec/                   # 🔑 RSPEC TESTS
+│   ├── spec_helper.rb      # Test configuration
+│   └── sec_api/            # Test files mirroring lib/ structure
+├── .gitignore
+├── .node-version           # Node version (for tooling)
+├── .rspec                  # RSpec configuration
+├── .rspec_status           # Test run status
+├── .standard.yml           # Standard Ruby linter configuration
+├── CHANGELOG.md            # Version history
+├── CLAUDE.md               # Claude session notes
+├── Gemfile                 # Gem dependencies
+├── Gemfile.lock            # Locked dependency versions
+├── LICENSE.txt             # MIT License
+├── README.md               # Project README
+├── Rakefile                # Rake tasks
+└── sec_api.gemspec         # Gem specification
+```
+
+## Critical Directories Explained
+
+### `/lib/sec_api/`  - Main Library Code
+
+**Purpose:** Core gem functionality organized by technical layer
+
+**Architecture Pattern:** Client → Proxy pattern with middleware stack
+
+**Key Components:**
+- **client.rb** - Main entry point, delegates to proxies
+- **errors/** - Complete exception hierarchy (TransientError/PermanentError)
+- **middleware/** - Faraday middleware (retry, rate limiting, error handling)
+- **objects/** - Immutable value objects (Dry::Struct)
+- **collections/** - Collection wrappers with Enumerable interface
+- **Proxies:** query.rb, mapping.rb, extractor.rb, xbrl.rb
+
+### `/spec/` - Test Suite
+
+**Purpose:** RSpec tests mirroring lib/ structure
+
+**Coverage:** >90% target for v1.0.0
+
+**Testing Strategy:**
+- VCR/WebMock cassettes for API integration tests
+- Shared examples for cross-cutting behavior (retry, pagination, rate limiting)
+- Unit tests for pure logic
+
+### `/_bmad-output/planning-artifacts/` - Planning Documents
+
+**Purpose:** Product requirements, architecture decisions, and epic breakdown
+
+**Key Files:**
+- **prd.md** - Product Requirements Document
+- **architecture.md** - Architectural decisions and patterns
+- **epics.md** - Epic and story breakdown for implementation
+
+### `/config/` - Configuration Files
+
+**Purpose:** YAML configuration and local overrides
+
+**Files:**
+- **secapi.yml** - Default configuration (API key, retry settings, etc.)
+- **secapi.local.yml** (gitignored) - Local environment overrides
+
+## Entry Points
+
+### Main Entry Point
+**File:** `lib/sec_api.rb`
+**Purpose:** Requires all library files, provides top-level namespace
+
+### Client Initialization
+**File:** `lib/sec_api/client.rb`
+**Usage:**
+```ruby
+client = SecApi::Client.new  # Auto-loads config from YAML
+client.query                  # Query API proxy
+client.mapping                # Mapping API proxy
+client.extractor              # Extractor API proxy
+client.xbrl                   # XBRL API proxy
+```
+
+### Development Console
+**File:** `bin/console`
+**Purpose:** Interactive IRB session with gem loaded for manual testing
+
+## Code Organization Patterns
+
+### Naming Conventions
+- **Modules/Classes:** `SecApi::` namespace, PascalCase
+- **Files:** snake_case matching class names
+- **Methods:** snake_case (Ruby standard)
+- **Constants:** SCREAMING_SNAKE_CASE
+
+### File-to-Class Mapping
+- `lib/sec_api/client.rb` → `SecApi::Client`
+- `lib/sec_api/errors/rate_limit_error.rb` → `SecApi::RateLimitError`
+- `lib/sec_api/objects/filing.rb` → `SecApi::Filing`
+
+### Dependencies
+- **External:** Faraday, anyway_config, dry-struct
+- **Internal:** Client → Proxies → Middleware → HTTP API
+
+## Integration Points
+
+### External API
+- **sec-api.io REST API** - All HTTP requests via Faraday
+- **Base URL:** `https://api.sec-api.io` (configurable)
+- **Authentication:** API key in headers
+
+### Configuration
+- **YAML files:** `config/secapi.yml`
+- **Environment variables:** `SECAPI_*` prefix
+- **Managed by:** anyway_config gem
+
+### Testing
+- **Framework:** RSpec
+- **Mocking:** VCR/WebMock for HTTP requests
+- **Linting:** Standard Ruby (standardrb)
+
+## Future Directories (Planned for v1.0.0)
+
+Based on the Architecture document, these directories will be added:
+
+- `lib/sec_api/proxies/` - Organized proxy objects
+- `lib/sec_api/middleware/retry_config.rb` - Enhanced retry configuration
+- `lib/sec_api/middleware/rate_limiter.rb` - Rate limiting middleware
+- `lib/sec_api/middleware/instrumentation.rb` - Observability hooks
+- `spec/support/shared_examples/` - Shared test behavior
+- `spec/fixtures/vcr_cassettes/` - VCR cassettes organized by proxy
+- `docs/examples/` - Usage examples (query, backfill, streaming)
+- `docs/migration-guide-v1.md` - v0.1.0 → v1.0.0 migration guide

data/lib/sec_api/callback_helper.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+require "json"
+
+module SecApi
+  # Shared helper methods for callback invocation and error handling.
+  #
+  # This module provides consistent callback error logging across all middleware
+  # and client code. All callback invocations should use this module's helpers
+  # to ensure consistent structured logging when callbacks fail.
+  #
+  # @example Including in a middleware class
+  #   class MyMiddleware < Faraday::Middleware
+  #     include SecApi::CallbackHelper
+  #
+  #     def call(env)
+  #       invoke_callback_safely("my_callback") do
+  #         @config.my_callback&.call(data: "value")
+  #       end
+  #     end
+  #   end
+  #
+  module CallbackHelper
+    # Logs callback errors to the configured logger with structured JSON format.
+    #
+    # @param callback_name [String] Name of the callback that failed
+    # @param error [Exception] The exception that was raised
+    # @param config [SecApi::Config, nil] Config object with logger (optional)
+    # @return [void]
+    def log_callback_error(callback_name, error, config = nil)
+      # Use instance variable @config if config not passed
+      cfg = config || (defined?(@config) ? @config : nil) || (defined?(@_config) ? @_config : nil)
+      return unless cfg&.logger
+
+      begin
+        cfg.logger.error do
+          {
+            event: "secapi.callback_error",
+            callback: callback_name,
+            error_class: error.class.name,
+            error_message: error.message
+          }.to_json
+        end
+      rescue
+        # Don't let logging errors break anything
+      end
+    end
+  end
+end