sumologic-query 1.3.3 → 1.3.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f674ac09efecc2c167df4d3853be9d9bd5c1c043bf6e0d574f068770e9979a8b
-  data.tar.gz: 4e43daf71638156ad4ac9f72b862daf501f4714037b3cee603aac1a6e8de1efb
+  metadata.gz: 83e78d35f43a7e12f98e3ed11d9d4b0ddd7fe4cf567c281fa4729c8873c583a0
+  data.tar.gz: 4cc242bc523b5ce0cf709ec9f9aceb44cb1fe5911582983ae80e3a49d00a175d
 SHA512:
-  metadata.gz: cc1c1a801d4c0dcf282ae281cc4da00eb125095b036f45489cda7c74638e4116fde5ad5c317896fb8e05988fd143887d7b470f31d02fba10426cc3c3c7bd78a0
-  data.tar.gz: 3b360acf874598ea9ca2513d1ddd507c617e970b94f79efb5029fc3a2e37e807b1f8aac2e3e9cbe95793ef0d74f210fd2ab13e9d7d3f0d25fad446014f6f46cb
+  metadata.gz: 7b9d63d998ff14e1b9db86dc982dd84b508f642c01a62883b7bdce997ec21aed3097477956776319f01b68da2e8ed967b713894ed8002353ab99ee9fd72bad85
+  data.tar.gz: '084c1c7db713d88f88979941d8b6a261f48c4bc533f3ce2ac04cb97ea3bae5fb0762889611dbfffd790740e0c7928acffb9e7df144c6e6d7b0fa129128d49e23'

data/CHANGELOG.md

@@ -6,6 +6,40 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and release notes are automatically generated from commit messages.
+## [1.3.5](https://github.com/patrick204nqh/sumologic-query/compare/v1.3.4...v1.3.5) (2025-11-19)
+
+### 🎉 New Features
+
+- add discover-sources command for dynamic source discovery from logs
+- implement rate limiting configuration and enhance documentation for querying options
+
+### 🔧 Refactoring
+
+- improve source discovery logic and enhance debugging output
+
+### 📚 Documentation
+
+- update tldr.md with enhanced search options and new commands for querying logs
+
+
+
+## [1.3.4](https://github.com/patrick204nqh/sumologic-query/compare/v1.3.3...v1.3.4) (2025-11-19)
+
+### 🎉 New Features
+
+- add time parsing utility and enhance CLI time options for flexible querying
+- enhance debug logging to include request headers for better traceability
+
+### 🐛 Bug Fixes
+
+- freeze regex for relative time parsing and improve error message formatting
+
+### 📚 Documentation
+
+- update README and examples to enhance time format usage and add new time format examples
+
+
+
 ## [1.3.3](https://github.com/patrick204nqh/sumologic-query/compare/v1.3.2...v1.3.3) (2025-11-17)
 
 ### 🎉 New Features

data/README.md

@@ -6,309 +6,166 @@
 [![Downloads](https://img.shields.io/gem/dt/sumologic-query.svg)](https://rubygems.org/gems/sumologic-query)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-## Why This Tool?
+## Features
 
-- **Minimal dependencies**: Uses only Ruby stdlib + Thor for CLI
-- **Fast queries**: Efficient polling and automatic pagination
-- **Simple interface**: Just query, get results, done
-- **Read-only**: No write operations, perfect for safe log access
-- **Modular architecture**: Clean separation of concerns (HTTP, Search, Metadata)
-- **Metadata support**: List collectors and sources alongside log queries
-
-All existing Ruby Sumo Logic gems are unmaintained (2-9 years dormant). This tool provides a fresh, minimal approach focused on querying logs and metadata.
+- **Simple time parsing** - Use `-1h`, `-30m`, `now` instead of timestamps
+- **Dynamic source discovery** - Find CloudWatch/ECS/Lambda sources from logs
+- **Interactive mode** - Explore logs with FZF fuzzy search
+- **Timezone support** - US, Australian, and IANA formats
+- **Fast & efficient** - Smart polling and pagination
+- **Read-only** - Safe log access with no write operations
 
 ## Installation
 
-### Via RubyGems
-
 ```bash
+# Via RubyGems
 gem install sumologic-query
-```
-
-### Via Homebrew
 
-```bash
+# Via Homebrew
 brew tap patrick204nqh/tap
 brew install sumologic-query
 ```
 
-### From Source
-
-```bash
-git clone https://github.com/patrick204nqh/sumologic-query.git
-cd sumologic-query
-bundle install
-bundle exec rake install
-```
-
 ## Quick Start
 
-### 1. Set Up Credentials
-
-Export your Sumo Logic API credentials:
+### 1. Set Credentials
 
 ```bash
 export SUMO_ACCESS_ID="your_access_id"
 export SUMO_ACCESS_KEY="your_access_key"
-export SUMO_DEPLOYMENT="us2"  # Optional: us1, us2 (default), eu, au, etc.
+export SUMO_DEPLOYMENT="us2"  # Optional: us1, us2 (default), eu, au
 ```
 
-**Getting credentials:**
-1. Log in to Sumo Logic
-2. Go to **Administration → Security → Access Keys**
-3. Create a new access key or use existing
-4. Copy the Access ID and Access Key
+Get credentials: Sumo Logic → **Administration → Security → Access Keys**
 
-### 2. Run Your First Query
+### 2. Run Queries
 
 ```bash
 # Search logs
-sumo-query search --query 'error' \
-  --from '2025-11-13T14:00:00' \
-  --to '2025-11-13T15:00:00' \
-  --limit 10
+sumo-query search -q 'error' -f '-1h' -t 'now' --limit 100
 
-# List collectors
-sumo-query collectors
+# Discover dynamic sources (CloudWatch/ECS/Lambda)
+sumo-query discover-sources
 
-# List sources
+# List collectors and sources
+sumo-query collectors
 sumo-query sources
 ```
 
-## Usage
-
-The CLI provides three main commands:
+## Commands
 
-### Search Logs
+### 1. Search Logs
 
 ```bash
-sumo-query search --query "YOUR_QUERY" \
-  --from "START_TIME" \
-  --to "END_TIME" \
-  [--output FILE] \
-  [--limit N] \
-  [--time-zone TZ] \
-  [--interactive]
+sumo-query search -q "YOUR_QUERY" -f "START" -t "END" [OPTIONS]
 ```
 
-**Required options:**
-- `-q, --query QUERY` - Sumo Logic query string
-- `-f, --from TIME` - Start time (ISO 8601 format)
-- `-t, --to TIME` - End time (ISO 8601 format)
+**Options:**
+- `-q, --query` - Query string (required)
+- `-f, --from` - Start time (required, e.g., `-1h`, `2025-11-19T14:00:00`)
+- `-t, --to` - End time (required, e.g., `now`)
+- `-z, --time-zone` - Timezone (default: UTC)
+- `-l, --limit` - Max messages to return
+- `-o, --output` - Save to file
+- `-i, --interactive` - Launch FZF browser
+- `-d, --debug` - Debug output
 
-**Optional options:**
-- `-i, --interactive` - Launch interactive browser with FZF
-- `-z, --time-zone TZ` - Time zone (default: UTC)
-- `-l, --limit N` - Limit number of messages
-- `-o, --output FILE` - Save to file (default: stdout)
-- `-d, --debug` - Enable debug output
+**Interactive Mode** (`-i`): FZF-based browser with fuzzy search, preview, and multi-select. Requires `fzf` ([install](https://github.com/junegunn/fzf#installation)).
 
-### Interactive Mode 🚀
-
-Explore your logs interactively with a powerful FZF-based interface:
+### 2. Discover Dynamic Sources
 
 ```bash
-# Launch interactive mode
-sumo-query search --query 'error' \
-  --from '2025-11-13T14:00:00' \
-  --to '2025-11-13T15:00:00' \
-  --interactive
-
-# Or use the shorthand
-sumo-query search -q 'error' -f '2025-11-13T14:00:00' -t '2025-11-13T15:00:00' -i
+sumo-query discover-sources [OPTIONS]
 ```
 
-**Features:**
-- 🔍 Fuzzy search across all message fields
-- 👁️ Live preview with full JSON details
-- 🎨 Color-coded log levels (ERROR, WARN, INFO)
-- ⌨️ Keyboard shortcuts for quick actions
-- 📋 Multi-select and batch operations
-- 💾 Export selected messages
-
-**Keybindings:**
-- `Enter` - Toggle selection (mark/unmark message)
-- `Tab` - Open current message in pager (copyable view)
-- `Ctrl-S` - Save selected messages to `sumo-selected.txt` and exit
-- `Ctrl-Y` - Copy selected messages to clipboard and exit
-- `Ctrl-E` - Export selected messages to `sumo-export.jsonl` and exit
-- `Ctrl-A` - Select all messages
-- `Ctrl-D` - Deselect all messages
-- `Ctrl-/` - Toggle preview pane
-- `Ctrl-Q` - Quit without saving
-
-**Requirements:**
-- Install FZF: `brew install fzf` (macOS) or `apt-get install fzf` (Linux)
-- See: https://github.com/junegunn/fzf#installation
-
-### List Collectors
-
-```bash
-sumo-query collectors [--output FILE]
-```
+Finds dynamic source names from log data (CloudWatch, ECS, Lambda streams).
 
-Lists all collectors in your account with status and metadata.
+**Options:**
+- `-f, --from` - Start time (default: `-24h`)
+- `-t, --to` - End time (default: `now`)
+- `--filter` - Filter query (e.g., `_sourceCategory=*ecs*`)
+- `-z, --time-zone` - Timezone (default: UTC)
+- `-o, --output` - Save to file
 
-### List Sources
+**Examples:**
 
 ```bash
-sumo-query sources [--output FILE]
-```
-
-Lists all sources from active collectors.
-
-**See [examples/queries.md](examples/queries.md) for more query patterns and examples.**
+# Discover all sources from last 24 hours
+sumo-query discover-sources
 
-## Ruby Library Usage
+# Filter to ECS only
+sumo-query discover-sources --filter '_sourceCategory=*ecs*'
 
-Use the library directly in your Ruby code:
-
-```ruby
-require 'sumologic'
-
-# Initialize client
-client = Sumologic::Client.new(
-  access_id: ENV['SUMO_ACCESS_ID'],
-  access_key: ENV['SUMO_ACCESS_KEY'],
-  deployment: 'us2'
-)
-
-# Search logs
-results = client.search(
-  query: 'error',
-  from_time: '2025-11-13T14:00:00',
-  to_time: '2025-11-13T15:00:00',
-  time_zone: 'UTC',
-  limit: 1000
-)
+# Last 7 days, save to file
+sumo-query discover-sources -f '-7d' -o sources.json
+```
 
-results.each do |message|
-  puts message['map']['message']
-end
+### 3. List Collectors & Sources
 
+```bash
 # List collectors
-collectors = client.list_collectors
+sumo-query collectors [-o FILE]
 
-# List all sources
-sources = client.list_all_sources
+# List static sources
+sumo-query sources [-o FILE]
 ```
 
-**See [docs/api-reference.md](docs/api-reference.md) for complete API documentation.**
-
 ## Time Formats
 
-Use ISO 8601 format for timestamps:
-
 ```bash
-# UTC timestamps (default)
---from "2025-11-13T14:30:00" --to "2025-11-13T15:00:00"
+# Relative (recommended)
+-1h, -30m, -7d, now
 
-# With timezone
---from "2025-11-13T14:30:00" --time-zone "America/New_York"
+# ISO 8601
+2025-11-19T14:00:00
 
-# Using shell helpers
---from "$(date -u -v-1H '+%Y-%m-%dT%H:%M:%S')"  # 1 hour ago
---to "$(date -u '+%Y-%m-%dT%H:%M:%S')"          # now
-```
+# Unix timestamp
+1700000000
 
-## Output Format
-
-Results are returned as JSON:
-
-```json
-{
-  "query": "error",
-  "from": "2025-11-13T14:00:00",
-  "to": "2025-11-13T15:00:00",
-  "time_zone": "UTC",
-  "message_count": 42,
-  "messages": [
-    {
-      "map": {
-        "_messagetime": "1731506400123",
-        "_sourceCategory": "prod/api",
-        "_sourceName": "api-server-01",
-        "message": "Error processing request: timeout"
-      }
-    }
-  ]
-}
+# Timezones
+UTC, AEST, EST, America/New_York, Australia/Sydney, +10:00
 ```
 
-## Performance
-
-Query execution time depends on data volume:
+See [examples/queries.md](examples/queries.md) for comprehensive query patterns.
 
-| Messages | Typical Time |
-|----------|--------------|
-| < 10K | 30-60 seconds |
-| 10K-100K | 1-2 minutes |
-| 100K+ | 2-5 minutes |
+## Ruby Library
 
-**Tips for faster queries:**
-- Narrow your time range
-- Add `_sourceCategory` filters
-- Use `--limit` to cap results
-- Use aggregation queries instead of raw messages
-
-## Documentation
-
-- **[Quick Reference (tldr)](docs/tldr.md)** - Concise command examples in tldr format
-- **[Query Examples](examples/queries.md)** - Common query patterns and use cases
-- **[API Reference](docs/api-reference.md)** - Complete Ruby library documentation
-- **[Architecture](docs/architecture/)** - System design and architecture decisions
-- **[Troubleshooting](docs/troubleshooting.md)** - Common issues and solutions
-
-## Development
+```ruby
+require 'sumologic'
 
-See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, testing, and contribution guidelines.
+client = Sumologic::Client.new(
+  access_id: ENV['SUMO_ACCESS_ID'],
+  access_key: ENV['SUMO_ACCESS_KEY']
+)
 
-Quick start:
+# Search
+client.search(query: 'error', from_time: '-1h', to_time: 'now')
 
-```bash
-# Clone and install
-git clone https://github.com/patrick204nqh/sumologic-query.git
-cd sumologic-query
-bundle install
+# Discover sources
+client.discover_dynamic_sources(from_time: '-24h', to_time: 'now')
 
-# Run tests
-bundle exec rspec
+# Metadata
+client.list_collectors
+client.list_all_sources
+```
 
-# Run linter
-bundle exec rubocop
+## Documentation
 
-# Test locally
-bundle exec bin/sumo-query search --query "error" \
-  --from "2025-11-13T14:00:00" \
-  --to "2025-11-13T15:00:00"
-```
+- [Query Examples](examples/queries.md) - Query patterns and examples
+- [Quick Reference](docs/tldr.md) - Command cheat sheet
+- [Rate Limiting](docs/rate-limiting.md) - Performance tuning
+- [Architecture](docs/architecture/) - Design decisions
 
 ## Contributing
 
-Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details.
-
-1. Fork the repository
-2. Create your feature branch (`git checkout -b feature/amazing-feature`)
-3. Commit your changes (`git commit -m 'Add amazing feature'`)
-4. Push to the branch (`git push origin feature/amazing-feature`)
-5. Open a Pull Request
+See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
 
 ## License
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
-
-## Support
-
-- **Issues**: [GitHub Issues](https://github.com/patrick204nqh/sumologic-query/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/patrick204nqh/sumologic-query/discussions)
-- **Documentation**: [docs/](docs/)
-
-## Resources
-
-- **Sumo Logic API Docs**: https://help.sumologic.com/docs/api/search-job/
-- **Query Language**: https://help.sumologic.com/docs/search/
-- **Bug Reports**: https://github.com/patrick204nqh/sumologic-query/issues
+MIT License - see [LICENSE](LICENSE) file.
 
----
+## Links
 
-**Note**: This tool provides read-only access to Sumo Logic logs. It does not modify any data or configuration.
+- [Issues](https://github.com/patrick204nqh/sumologic-query/issues)
+- [Sumo Logic API Docs](https://help.sumologic.com/docs/api/search-job/)
+- [Query Language](https://help.sumologic.com/docs/search/)

data/lib/sumologic/cli/commands/base_command.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'fileutils'
+
 module Sumologic
   class CLI < Thor
     module Commands
@@ -19,6 +21,10 @@ module Sumologic
           json_output = JSON.pretty_generate(data)
 
           if options[:output]
+            # Create parent directories if they don't exist
+            output_dir = File.dirname(options[:output])
+            FileUtils.mkdir_p(output_dir) unless output_dir == '.'
+
             File.write(options[:output], json_output)
             warn "\nResults saved to: #{options[:output]}"
           else

data/lib/sumologic/cli/commands/discover_sources_command.rb

@@ -0,0 +1,59 @@
+# frozen_string_literal: true
+
+require_relative 'base_command'
+require_relative '../../utils/time_parser'
+
+module Sumologic
+  class CLI < Thor
+    module Commands
+      # Handles the discover-sources command execution
+      class DiscoverSourcesCommand < BaseCommand
+        def execute
+          parse_time_options
+          log_discovery_info
+          results = perform_discovery
+
+          output_json(results)
+        end
+
+        private
+
+        def parse_time_options
+          # Parse time formats and store both original and parsed values
+          @original_from = options[:from]
+          @original_to = options[:to]
+          @parsed_from = Utils::TimeParser.parse(options[:from])
+          @parsed_to = Utils::TimeParser.parse(options[:to])
+          @parsed_timezone = Utils::TimeParser.parse_timezone(options[:time_zone])
+        rescue Utils::TimeParser::ParseError => e
+          warn "Error parsing time: #{e.message}"
+          exit 1
+        end
+
+        def log_discovery_info
+          warn '=' * 60
+          warn 'Discovering Dynamic Source Names'
+          warn '=' * 60
+          warn "Time Range: #{@original_from} to #{@original_to}"
+          if @original_from != @parsed_from || @original_to != @parsed_to
+            warn "  (Parsed: #{@parsed_from} to #{@parsed_to})"
+          end
+          warn "Time Zone: #{@parsed_timezone}"
+          warn "Filter: #{options[:filter] || 'none (all sources)'}"
+          warn '-' * 60
+          warn 'Running aggregation query to discover sources...'
+          $stderr.puts
+        end
+
+        def perform_discovery
+          client.discover_dynamic_sources(
+            from_time: @parsed_from,
+            to_time: @parsed_to,
+            time_zone: @parsed_timezone,
+            filter: options[:filter]
+          )
+        end
+      end
+    end
+  end
+end

data/lib/sumologic/cli/commands/search_command.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require_relative 'base_command'
+require_relative '../../utils/time_parser'
 
 module Sumologic
   class CLI < Thor
@@ -8,6 +9,7 @@ module Sumologic
       # Handles the search command execution
       class SearchCommand < BaseCommand
         def execute
+          parse_time_options
           log_search_info
           results = perform_search
 
@@ -22,11 +24,26 @@ module Sumologic
 
         private
 
+        def parse_time_options
+          # Parse time formats and store both original and parsed values
+          @original_from = options[:from]
+          @original_to = options[:to]
+          @parsed_from = Utils::TimeParser.parse(options[:from])
+          @parsed_to = Utils::TimeParser.parse(options[:to])
+          @parsed_timezone = Utils::TimeParser.parse_timezone(options[:time_zone])
+        rescue Utils::TimeParser::ParseError => e
+          warn "Error parsing time: #{e.message}"
+          exit 1
+        end
+
         def log_search_info
           warn '=' * 60
           warn 'Sumo Logic Search Query'
           warn '=' * 60
-          warn "Time Range: #{options[:from]} to #{options[:to]}"
+          warn "Time Range: #{@original_from} to #{@original_to}"
+          if @original_from != @parsed_from || @original_to != @parsed_to
+            warn "  (Parsed: #{@parsed_from} to #{@parsed_to})"
+          end
           warn "Query: #{options[:query]}"
           warn "Limit: #{options[:limit] || 'unlimited'}"
           warn '-' * 60
@@ -37,9 +54,9 @@ module Sumologic
         def perform_search
           client.search(
             query: options[:query],
-            from_time: options[:from],
-            to_time: options[:to],
-            time_zone: options[:time_zone],
+            from_time: @parsed_from,
+            to_time: @parsed_to,
+            time_zone: @parsed_timezone,
             limit: options[:limit]
           )
         end
@@ -54,9 +71,11 @@ module Sumologic
         def output_search_results(results)
           output_json(
             query: options[:query],
-            from: options[:from],
-            to: options[:to],
-            time_zone: options[:time_zone],
+            from: @parsed_from,
+            to: @parsed_to,
+            from_original: @original_from,
+            to_original: @original_to,
+            time_zone: @parsed_timezone,
             message_count: results.size,
             messages: results
           )
@@ -75,9 +94,9 @@ module Sumologic
         def build_formatted_results(results)
           {
             'query' => options[:query],
-            'from' => options[:from],
-            'to' => options[:to],
-            'time_zone' => options[:time_zone],
+            'from' => @parsed_from,
+            'to' => @parsed_to,
+            'time_zone' => @parsed_timezone,
             'message_count' => results.size,
             'messages' => results
           }