firecrawl 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3ba3c8f728651fdc912a44ad7aee4fcdae849cbac7f351b62dc8cec9fdb29973
-  data.tar.gz: dfd0382fb5cafdd471572cb1e3c470b7c28548d20a466c469bbec53ea5c41209
+  metadata.gz: 7e55dc5e433f0632ab0c11feda818bf82ddf77ae8ea2fdaa624c5a9af1dccf4c
+  data.tar.gz: 2781378e0a6b62c2e7befb0ebd0298b0b91b2cf249fe33115dc675606aed7bfe
 SHA512:
-  metadata.gz: 03a76beec314251ed927c8abc6a2fc2493c464dff7907699d484c752b3a153e25a9df184f03f98d7849bf944051010807b7dbb549a1b466710a3b22e83cdd96c
-  data.tar.gz: ca04e151e6ffc27e38325fe1a4ae701fc007cee62659b77ceb079191b4c07667b91accd38dcec5531a194a5c7b9f0d685b4a26e0ee8d9679792040ac07745f6b
+  metadata.gz: 6fa88114c36df02f9cd261159132298e9a44b01464334daeb31ea2d8d5b11122321066ddf55207fddc3bef72704b353cf42d7aebaa46ac70298ea1efe19c6885
+  data.tar.gz: 622fd277c01854131b4a21742c915359c97fffa54b4dc41aff47d09b06d4d7c6c485361ff6409dc1361cf561f313d42096674a426987b5bbf696dcba1f52cd96

data/README.md

@@ -11,7 +11,7 @@ provide markdown information for real time information lookup as well as groundi
 require 'firecrawl'
 
 Firecrawl.api_key ENV[ 'FIRECRAWL_API_KEY' ]
-response = Firecrawl.scrape( 'https://example.com', options )
+response = Firecrawl.scrape( 'https://example.com' )
 if response.success?
   result = response.result 
   puts result.metadata[ 'title' ]
@@ -45,18 +45,44 @@ $ gem install firecrawl
 
 ## Usage
 
-### Basic Scraping
+### Scraping
 
-The simplest way to use Firecrawl is to scrape a single page:
+The simplest way to use Firecrawl is to `scrape`, which will scrape the content of a single page
+at the given url and optionally convert it to markdown as well as create a screenshot. You can 
+chose to scrape the entire page or only the main content.
 
 ```ruby
-Firecrawl.api_key ENV['FIRECRAWL_API_KEY']
-response = Firecrawl.scrape('https://example.com', format: :markdown )
+Firecrawl.api_key ENV[ 'FIRECRAWL_API_KEY' ]
+response = Firecrawl.scrape( 'https://example.com', format: :markdown )
 
 if response.success?
   result = response.result
   if result.success?
-    puts result.metadata['title']
+    puts result.metadata[ 'title' ]
+    puts result.markdown
+  end
+else
+  puts response.result.error_description
+end
+```
+
+In this basic example we have globally set the `Firecrawl.api_key` from the environment and then
+used the `Firecrawl.scrape` convenience method to make a request to the Firecrawl API to scrape 
+the `https://example.com` page and return markdown ( markdown and the main content of the page 
+are returned by default so we could have ommitted the options entirelly ).
+
+The `Firecrawl.scrape` method instantiates a `Firecrawl::ScrapeRequest` instance and then calls
+it's `submit` method. The following is the equivalent code which makes explict use of the 
+`Firecrawl::ScrapeRequest` class.
+
+```ruby
+request = Firecrawl::ScrapeRequest.new( api_key: ENV[ 'FIRECRAWL_API_KEY' ] )
+response = request.submit( 'https://example.com', format: :markdown )
+
+if response.success?
+  result = response.result
+  if result.success?
+    puts result.metadata[ 'title' ]
     puts result.markdown
   end
 else
@@ -64,9 +90,14 @@ else
 end
 ```
 
-### Scrape Options
+Notice also that in this example we've directly passed the `api_key` to the individual request. 
+This is optional. If you set the key globally and omit it in the request constructor the 
+`ScrapeRequest` instance will use the globally assigned `api_key`.
+
+#### Scrape Options
 
-You can customize scraping behavior using `ScrapeOptions`:
+You can customize scraping behavior using options, either by passing an option hash to 
+`submit` method, as we have done above, or by building a `ScrapeOptions` instance:
 
 ```ruby
 options = Firecrawl::ScrapeOptions.build do 
@@ -78,9 +109,28 @@ options = Firecrawl::ScrapeOptions.build do
 end
 
 request = Firecrawl::ScrapeRequest.new( api_key: ENV[ 'FIRECRAWL_API_KEY' ] )
-response = request.scrape('https://example.com', options)
+response = request.submit( 'https://example.com', options )
 ```
 
+#### Scrape Response
+
+The `Firecrawl` gem is based on the `Faraday` gem, which permits you to customize the request
+orchestration, up to and including changing the actual HTTP implementation used to make the 
+request. See Connections below for additional details.
+
+Any `Firecrawl` request, including the `submit` method as used above, will thus return a 
+`Faraday::Response`. This response includes a `success?` method which indicates if the request 
+was successful. If the request was successful, the `response.result` method will be an instance 
+of `Firecrawl::ScrapeResult` that will encapsulate the scraping result. This instance, in turn, 
+has a `success?` method which will return `true` if Firecrawl successfully scraped the page. 
+
+A successful result will include html, markdown, screenshot, as well as any action and llm 
+results and related metadata. 
+
+If the response is not successful ( if `response.success?` is `false` ) then `response.result` 
+will be an instance of Firecrawl::ErrorResult which will provide additional details about the 
+nature of the failure.
+
 ### Batch Scraping
 
 For scraping multiple URLs efficiently:
@@ -94,7 +144,7 @@ options = Firecrawl::ScrapeOptions.build do
   only_main_content true
 end
 
-response = request.scrape( urls, options )
+response = request.submit( urls, options )
 while response.success?
   batch_result = response.result
   batch_result.scrape_results.each do |result|
@@ -104,7 +154,7 @@ while response.success?
   end
   break unless batch_result.status?( :scraping )
   sleep 0.5
-  response = request.retrieve_scrape_results( batch_result )
+  response = request.retrieve( batch_result )
 end
 ```
 
@@ -120,7 +170,7 @@ options = Firecrawl::MapOptions.build do
   ignore_subdomains true
 end
 
-response = request.map( 'https://example.com', options )
+response = request.submit( 'https://example.com', options )
 if response.success?
   result = response.result
   result.links.each do |link|
@@ -145,51 +195,16 @@ options = Firecrawl::CrawlOptions.build do
   end
 end
 
-response = request.crawl( 'https://example.com', options )
+response = request.submit( 'https://example.com', options )
 while response.success?
   crawl_result = response.result
-  crawl_result.scrape_results.each do |result|
-    puts result.metadata['title']
+  crawl_result.scrape_results.each do | result |
+    puts result.metadata[ 'title' ]
     puts result.markdown
   end
-  break unless crawl_result.status?(:scraping)
+  break unless crawl_result.status?( :scraping )
   sleep 0.5
-  response = request.retrieve_crawl_results(crawl_result)
-end
-```
-
-## Response Structure
-
-All Firecrawl requests return a Faraday response with an added `result` method. The result will 
-be one of:
-
-- `ScrapeResult`: Contains the scraped content and metadata
-- `BatchScrapeResult`: Contains multiple scrape results
-- `MapResult`: Contains discovered links
-- `CrawlResult`: Contains scrape results from crawled pages
-- `ErrorResult`: Contains error information if the request failed
-
-### Working with Results
-
-```ruby
-response = request.scrape(url, options)
-if response.success?
-  result = response.result
-  if result.success?
-    # Access scraped content
-    puts result.metadata['title']
-    puts result.markdown
-    puts result.html
-    puts result.raw_html
-    puts result.screenshot_url
-    puts result.links
-    
-    # Check for warnings
-    puts result.warning if result.warning
-  end
-else
-  error = response.result
-  puts "#{error.error_type}: #{error.error_description}"
+  response = request.retrieve( crawl_result )
 end
 ```
 

data/firecrawl.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do | spec |
 
   spec.name             = 'firecrawl'
-  spec.version          = '0.1.0'
+  spec.version          = '0.2.0'
   spec.authors          = [ 'Kristoph Cichocki-Romanov' ]
   spec.email            = [ 'rubygems.org@kristoph.net' ]
 

data/lib/firecrawl/batch_scrape_request.rb

@@ -3,8 +3,8 @@ module Firecrawl
   ##
   # The +BatchScrapeRequest+ class encapsulates a batch scrape request to the Firecrawl API. 
   # After creating a new +BatchScrapeRequest+ instance you can begin batch scraping by calling 
-  # the +scrape+ method and then subsequently retrieve the results by calling the 
-  # +retrieve_scrape_results' method.
+  # the +submit+ method and then subsequently retrieve the results by calling the 
+  # +retrieve' method.
   #
   # === examples
   # 
@@ -18,7 +18,7 @@ module Firecrawl
   #   only_main_content     true
   # end
   # 
-  # batch_response = request.scrape( urls, options )
+  # batch_response = request.submit( urls, options )
   # while response.success?
   #   batch_result = batch_response.result 
   #   if batch_result.success?
@@ -30,7 +30,7 @@ module Firecrawl
   #     end
   #   end
   #   break unless batch_result.status?( :scraping )
-  #   batch_response = request.retrieve_scrape_results( batch_result )
+  #   batch_response = request.retrieve( batch_result )
   # end
   #
   # unless batch_response.success? 
@@ -40,7 +40,7 @@ module Firecrawl
   class BatchScrapeRequest < Request 
 
     ## 
-    # The +scrape+ method makes a Firecrawl '/batch/scrape/{id}' POST request which will initiate 
+    # The +submit+ method makes a Firecrawl '/batch/scrape/{id}' POST request which will initiate 
     # batch scraping of the given urls. 
     # 
     # The response is always an instance of +Faraday::Response+. If +response.success?+ is true,
@@ -51,7 +51,7 @@ module Firecrawl
     # successful and then +response.result.success?+ to validate that the API processed the
     # request successfuly. 
     #
-    def scrape( urls, options = nil, &block )        
+    def submit( urls, options = nil, &block )        
       if options
         options = options.is_a?( ScrapeOptions ) ? options : ScrapeOptions.build( options.to_h ) 
         options = options.to_h
@@ -61,23 +61,23 @@ module Firecrawl
       options[ :urls ] = [ urls ].flatten
       response = post( "#{BASE_URI}/batch/scrape", options, &block )
       result = nil 
+      attributes = JSON.parse( response.body, symbolize_names: true ) rescue nil
       if response.success?
-        attributes = ( JSON.parse( response.body, symbolize_names: true ) rescue nil )
         attributes ||= { success: false, status: :failed  }
         result = BatchScrapeResult.new( attributes[ :success ], attributes )
       else 
-        result = ErrorResult.new( response.status, attributes )
+        result = ErrorResult.new( response.status, attributes || {} )
       end
 
       ResponseMethods.install( response, result )  
     end 
 
     ## 
-    # The +retrieve_scrape_results+ method makes a Firecrawl '/batch/scrape' GET request which 
-    # will return the scrape results that were completed since the previous call to this method
-    # ( or, if this is the first call to this method, since the batch scrape was started ). Note 
-    # that there is no guarantee that there are any new batch scrape results at the time you make 
-    # this call ( scrape_results may be empty ).
+    # The +retrieve+ method makes a Firecrawl '/batch/scrape' GET request which will return the 
+    # scrape results that were completed since the previous call to this method ( or, if this is
+    # the first call to this method, since the batch scrape was started ). Note that there is no 
+    # guarantee that there are any new batch scrape results at the time you make this call 
+    # ( scrape_results may be empty ).
     # 
     # The response is always an instance of +Faraday::Response+. If +response.success?+ is +true+, 
     # then +response.result+ will be an instance +BatchScrapeResult+. If the request is not 
@@ -87,17 +87,53 @@ module Firecrawl
     # successful and then +response.result.success?+ to validate that the API processed the
     # request successfuly. 
     #
-    def retrieve_scrape_results( batch_result, &block )
+    def retrieve( batch_result, &block )
       raise ArgumentError, "The first argument must be an instance of BatchScrapeResult." \
         unless batch_result.is_a?( BatchScrapeResult )
       response = get( batch_result.next_url, &block )  
       result = nil 
+      attributes = JSON.parse( response.body, symbolize_names: true ) rescue nil
       if response.success? 
-        attributes = ( JSON.parse( response.body, symbolize_names: true ) rescue nil )
         attributes ||= { success: false, status: :failed  }
         result = batch_result.merge( attributes  )
       else 
-        result = ErrorResult.new( response.status, attributes )
+        result = ErrorResult.new( response.status, attributes || {} )
+      end 
+
+      ResponseMethods.install( response, result )     
+    end 
+
+    ## 
+    # The +retrieve_all+ method makes a Firecrawl '/batch/scrape' GET request which will return 
+    # the scrape results that were completed at the time of this call. Repeated calls to this 
+    # method will retrieve the scrape results previouslly returned as well as any scrape results 
+    # that have accumulated since.
+    #
+    # Note that there is no guarantee that there are any new batch scrape results at the time you 
+    # make this call ( scrape_results may be empty ).
+    # 
+    # The response is always an instance of +Faraday::Response+. If +response.success?+ is +true+, 
+    # then +response.result+ will be an instance +BatchScrapeResult+. If the request is not 
+    # successful then +response.result+ will be an instance of +ErrorResult+.
+    #
+    # Remember that you should call +response.success?+ to valida that the call to the API was
+    # successful and then +response.result.success?+ to validate that the API processed the
+    # request successfuly. 
+    #
+    def retrieve_all( batch_result, &block )
+      raise ArgumentError, "The first argument must be an instance of BatchScrapeResult." \
+        unless batch_result.is_a?( BatchScrapeResult )
+      response = get( batch_result.url, &block )  
+      result = nil 
+      attributes = JSON.parse( response.body, symbolize_names: true ) rescue nil
+      if response.success? 
+        attributes ||= { success: false, status: :failed  }
+        # the next url should not be set by this method so that retrieve and retrieve_all do 
+        # not impact each other 
+        attributes.delete( :next )
+        result = batch_result.merge( attributes  )
+      else 
+        result = ErrorResult.new( response.status, attributes || {} )
       end 
 
       ResponseMethods.install( response, result )     

data/lib/firecrawl/crawl_request.rb

@@ -2,9 +2,10 @@ module Firecrawl
 
   ##
   # The +CrawlRequest+ class encapsulates a crawl request to the Firecrawl API. After creating 
-  # a new +CrawlRequest+ instance you can begin crawling by calling the +crawl+ method and 
-  # then subsequently retrieving the results by calling the +retrieve_crawl_results+ method.
-  # You can also optionally cancel the crawling operation by calling +cancel_crawl+.
+  # a new +CrawlRequest+ instance you can begin crawling by calling the +submit+ method and 
+  # then subsequently retrieving the results by calling the +retrieve+ method.
+  #
+  # You can also optionally cancel the crawling operation by calling +cancel+.
   #
   # === examples
   # 
@@ -19,7 +20,7 @@ module Firecrawl
   #   end 
   # end
   # 
-  # crawl_response = request.crawl( urls, options )
+  # crawl_response = request.submit( urls, options )
   # while crawl_response.success?
   #   crawl_result = crawl_response.result 
   #   if crawl_result.success?
@@ -31,7 +32,7 @@ module Firecrawl
   #     end
   #   end
   #   break unless crawl_result.status?( :scraping )
-  #   crawl_response = request.
+  #   crawl_response = request.retrieve( crawl_result )
   # end
   #
   # unless crawl_response.success? 
@@ -41,7 +42,7 @@ module Firecrawl
   class CrawlRequest < Request 
 
     ## 
-    # The +crawl+ method makes a Firecrawl '/crawl' POST request which will initiate crawling 
+    # The +submit+ method makes a Firecrawl '/crawl' POST request which will initiate crawling 
     # of the given url. 
     # 
     # The response is always an instance of +Faraday::Response+. If +response.success?+ is true,
@@ -52,18 +53,18 @@ module Firecrawl
     # successful and then +response.result.success?+ to validate that the API processed the
     # request successfuly. 
     #
-    def crawl( url, options = nil, &block )        
+    def submit( url, options = nil, &block )        
       if options
         options = options.is_a?( CrawlOptions ) ? options : CrawlOptions.build( options.to_h ) 
         options = options.to_h
       else 
         options = {}
       end
-      options[ url ] = url
+      options[ :url ] = url
       response = post( "#{BASE_URI}/crawl", options, &block )
       result = nil 
+      attributes = JSON.parse( response.body, symbolize_names: true ) rescue nil
       if response.success?
-        attributes = ( JSON.parse( response.body, symbolize_names: true ) rescue nil )
         attributes ||= { success: false, status: :failed  }
         result = CrawlResult.new( attributes[ :success ], attributes )
       else 
@@ -74,10 +75,10 @@ module Firecrawl
     end 
 
     ## 
-    # The +retrieve_crawl_results+ method makes a Firecrawl '/crawl/{id}' GET request which 
-    # will return the crawl results that were completed since the previous call to this method
-    # ( or, if this is the first call to this method, since the crawl was started ). Note that 
-    # there is no guarantee that there are any new crawl results at the time you make this call 
+    # The +retrieve+ method makes a Firecrawl '/crawl/{id}' GET request which will return the 
+    # crawl results that were completed since the previous call to this method( or, if this is 
+    # the first call to this method, since the crawl was started ). Note that there is no 
+    # guarantee that there are any new crawl results at the time you make this call 
     # ( scrape_results may be empty ).
     # 
     # The response is always an instance of +Faraday::Response+. If +response.success?+ is 
@@ -88,25 +89,24 @@ module Firecrawl
     # successful and then +response.result.success?+ to validate that the API processed the
     # request successfuly. 
     #
-    def retrieve_crawl_results( crawl_result, &block )
+    def retrieve( crawl_result, &block )
       raise ArgumentError, "The first argument must be an instance of CrawlResult." \
         unless crawl_result.is_a?( CrawlResult )
       response = get( crawl_result.next_url, &block )  
       result = nil 
+      attributes = JSON.parse( response.body, symbolize_names: true ) rescue nil
       if response.success? 
-        attributes = ( JSON.parse( response.body, symbolize_names: true ) rescue nil )
-        attributes ||= { success: false, status: :failed  }
-        result = crawl_result.merge( attributes  )
+        result = crawl_result.merge( attributes || { success: false, status: :failed  } )
       else 
-        result = ErrorResult.new( response.status, attributes )
+        result = ErrorResult.new( response.status, attributes || {} )
       end 
 
       ResponseMethods.install( response, result )     
     end
 
     ## 
-    # The +cance_crawl+ method makes a Firecrawl '/crawl/{id}' DELETE request which will cancel 
-    # a previouslly started crawl.
+    # The +cancel+ method makes a Firecrawl '/crawl/{id}' DELETE request which will cancel a 
+    # previouslly submitted crawl.
     # 
     # The response is always an instance of +Faraday::Response+. If +response.success?+ is 
     # +true+, then +response.result+ will be an instance +CrawlResult+. If the request is not 
@@ -116,17 +116,16 @@ module Firecrawl
     # successful and then +response.result.success?+ to validate that the API processed the
     # request successfuly. 
     #
-    def cancel_crawl( crawl_result, &block )
+    def cancel( crawl_result, &block )
       raise ArgumentError, "The first argument must be an instance of CrawlResult." \
         unless crawl_result.is_a?( CrawlResult )
       response = get( crawl_result.url, &block )  
       result = nil 
+      attributes = JSON.parse( response.body, symbolize_names: true ) rescue nil
       if response.success? 
-        attributes = ( JSON.parse( response.body, symbolize_names: true ) rescue nil )
-        attributes ||= { success: false, status: :failed  }
-        result = crawl_result.merge( attributes  )
+        result = crawl_result.merge( attributes || { success: false, status: :failed  } )
       else 
-        result = ErrorResult.new( response.status, attributes )
+        result = ErrorResult.new( response.status, attributes || {} )
       end 
 
       ResponseMethods.install( response, result )     

data/lib/firecrawl/error_result.rb

@@ -5,7 +5,7 @@ module Firecrawl
     
     def initialize( status_code, attributes = nil )
       @error_code, @error_description = status_code_to_error( status_code )
-      @error_description = attributes[ :error ] if @attributes&.respond_to?( :[] )
+      @error_description = attributes[ :error ] if attributes&.respond_to?( :[] )
     end 
 
   private 

data/lib/firecrawl/map_request.rb

@@ -10,7 +10,7 @@ module Firecrawl
   #
   # request = Firecrawl::MapRequest.new( api_key: ENV[ 'FIRECRAWL_API_KEY' )
   #
-  # response = request.map( 'https://example.com', { limit: 100 } )
+  # response = request.submit( 'https://example.com', { limit: 100 } )
   # if response.success?
   #   result = response.result 
   #   if result.success? 
@@ -25,14 +25,14 @@ module Firecrawl
   class MapRequest < Request 
 
     ## 
-    # The +map+ method makes a Firecrawl '/map' POST request which will scrape the site with 
-    # given url. 
+    # The +submit+ method makes a Firecrawl '/map' POST request which will scrape the site with 
+    # given url and return links to all hosted pages related to that url. 
     # 
     # The response is always an instance of +Faraday::Response+. If +response.success?+ is true,
     # then +response.result+ will be an instance +MapResult+. If the request is not successful 
     # then +response.result+ will be an instance of +ErrorResult+.
     #
-    def map( url, options = nil, &block )    
+    def submit( url, options = nil, &block )    
       if options
         options = options.is_a?( MapOptions ) ? options : MapOptions.build( options.to_h ) 
         options = options.to_h

data/lib/firecrawl/module_methods.rb

@@ -12,7 +12,7 @@ module Firecrawl
     end  
 
     def scrape( url, options = nil, &block )
-      Firecrawl::ScrapeRequest.new.scrape( url, options, &block ) 
+      Firecrawl::ScrapeRequest.new.submit( url, options, &block ) 
     end  
   end
 end

data/lib/firecrawl/scrape_request.rb

@@ -1,7 +1,7 @@
 module Firecrawl 
   ##
   # The +ScrapeRequest+ class encapsulates a '/scrape' POST request to the Firecrawl API. After 
-  # creating a new +ScrapeRequest+ instance you can initiate the request by calling the +scrape+ 
+  # creating a new +ScrapeRequest+ instance you can initiate the request by calling the +submit+ 
   # method to perform synchronous scraping.  
   #
   # === examples
@@ -15,7 +15,7 @@ module Firecrawl
   #   only_main_content     true
   # end
   #
-  # response = request.scrape( 'https://example.com', options )
+  # response = request.submit( 'https://example.com', options )
   # if response.success?
   #   result = response.result 
   #   puts response.metadata[ 'title ] 
@@ -28,13 +28,13 @@ module Firecrawl
   class ScrapeRequest < Request 
 
     ## 
-    # The +scrape+ method makes a Firecrawl '/scrape' POST request which will scrape the given url. 
+    # The +submit+ method makes a Firecrawl '/scrape' POST request which will scrape the given url. 
     # 
     # The response is always an instance of +Faraday::Response+. If +response.success?+ is true,
     # then +response.result+ will be an instance +ScrapeResult+. If the request is not successful 
     # then +response.result+ will be an instance of +ErrorResult+.
     #
-    def scrape( url, options = nil, &block )    
+    def submit( url, options = nil, &block )    
       if options
         options = options.is_a?( ScrapeOptions ) ? options : ScrapeOptions.build( options.to_h ) 
         options = options.to_h

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: firecrawl
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Kristoph Cichocki-Romanov
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-11-06 00:00:00.000000000 Z
+date: 2024-11-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: faraday