curlyq 0.0.8 → 0.0.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d3e32b382d7318b067ee3fb22f2e9057cf6aa9facfac41c74a0ebb5d4fb4743d
-  data.tar.gz: d379da3f0db621052e61230356f5c58b587eefccbb0a4c997216516a4159b44a
+  metadata.gz: 44e01914de08789721522e24e506fe88a49106610fac9a16736efcba0916be88
+  data.tar.gz: ec2887fee0dab67c64c0095f59091e867445b22559674b31f2eea64d8f4b9fea
 SHA512:
-  metadata.gz: ae63654deb943771e5f6f3aa0f6a037b1015336abbd696a8ce77acc22f361a3b6a18b03f3b7d02e5c7d5dcaa8d3608248bed240679acfce22ba2e462d84b529f
-  data.tar.gz: 481f8499e45a65cb3981fcf20ef7fc9f01f97a1b7014c6566aa2f3bf7a6611fd2d5d35f78e742e4063eea192b938c0642f0ca764e5032f330778d2815a191a41
+  metadata.gz: aa8338482e3d9414e6347195abc7ba8645adbc120f13469f219df7f2aa5fa1ba3c209740c608f728539969502965f6c29afc9d65b5ba411c8387b26ebd640c9d
+  data.tar.gz: 4fe071cb872a259795163da084851afdb5d003f7607a82a5ab6fe868f7f2edae22f0caa31327dfe1bad31e28e48fddde1857d882344359db8bf47b8579aef22c

data/CHANGELOG.md

@@ -1,3 +1,13 @@
+### 0.0.9
+
+2024-01-16 12:38
+
+#### IMPROVED
+
+- You can now use dot syntax inside of a square bracket comparison in --query (`[attrs.id*=what]`)
+- *=, ^=, $=, and == work with array values
+- [] comparisons with no comparison, e.g. [attrs.id], will return every match that has that element populated
+
 ### 0.0.8
 
 2024-01-15 16:45

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    curlyq (0.0.8)
+    curlyq (0.0.9)
       gli (~> 2.21.0)
       nokogiri (~> 1.16.0)
       selenium-webdriver (~> 4.16.0)

data/README.md

@@ -10,10 +10,13 @@ _If you find this useful, feel free to [buy me some coffee][donate]._
 [donate]: https://brettterpstra.com/donate
 
 
-The current version of `curlyq` is 0.0.8
+[jq]: https://github.com/jqlang/jq "Command-line JSON processor"
+[yq]: https://github.com/mikefarah/yq "yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor"
+
+The current version of `curlyq` is 0.0.9
 .
 
-CurlyQ is a utility that provides a simple interface for curl, with additional features for things like extracting images and links, finding elements by CSS selector or XPath, getting detailed header info, and more. It's designed to be part of a scripting pipeline, outputting everything as structured data (JSON or YAML). It also has rudimentary support for making calls to JSON endpoints easier, but it's expected that you'll use something like `jq` to parse the output.
+CurlyQ is a utility that provides a simple interface for curl, with additional features for things like extracting images and links, finding elements by CSS selector or XPath, getting detailed header info, and more. It's designed to be part of a scripting pipeline, outputting everything as structured data (JSON or YAML). It also has rudimentary support for making calls to JSON endpoints easier, but it's expected that you'll use something like [jq] to parse the output.
 
 [github]: https://github.com/ttscoff/curlyq/
 
@@ -44,7 +47,7 @@ SYNOPSIS
     curlyq [global options] command [command options] [arguments...]
 
 VERSION
-    0.0.8
+    0.0.9
 
 GLOBAL OPTIONS
     --help          - Show this message
@@ -71,6 +74,9 @@ You can shape the results using `--search` (`-s`) and `--query` (`-q`) on some c
 
 A search uses either CSS or XPath syntax to locate elements. For example, if you wanted to locate all of the `<article>` elements with a class of `post` inside of the div with an id of `main`, you would run `--search '#main article.post'`. Searches can target tags, ids, and classes, and can accept `>` to target direct descendents. You can also use XPaths, but I hate those so I'm not going to document them.
 
+> I've tried to make the query function useful, but if you want to do any kind of advanced shaping, you're better off piping the JSON output to [jq] or [yq].
+
+
 Queries are specifically for shaping CurlyQ output. If you're using the `html` command, it returns a key called `images`, so you can target just the images in the response with `-q 'images'`. The queries accept array syntax, so to get the first image, you would use `-q 'images[0]'`. Ranges are accepted as well, so `-q 'images[1..4]'` will return the 2nd through 5th images found on the page. You can also do comparisons, e.g. `images[rel=me]'` to target only images with a `rel` attribute of `me`.
 
 The comparisons for the query flag are:
@@ -84,6 +90,16 @@ The comparisons for the query flag are:
 - `^=` starts with text
 - `$=` ends with text
 
+Comparisons can be numeric or string comparisons. A numeric comparison like `curlyq images -q '[width>500]' URL` would return all of the images on the page with a width attribute greater than 500.
+
+You can also use dot syntax inside of comparisons, e.g. `[links.rel*=me]` to target the links object (`html` command), and return only the links with a `rel=me` attribute. If the comparison is to an array object (like `class` or `rel`), it will match if any of the elements of the array match your comparison.
+
+If you end the query with a specific key, only that key will be output. If there's only one match, it will be output as a raw string. If there are multiple matches, output will be an array:
+
+    curlyq tags --search '#main .post h3' -q '[attrs.id*=what].source' 'https://brettterpstra.com/2024/01/10/introducing-curlyq-a-pipeline-oriented-curl-helper/'
+    
+    <h3 id="whats-next">What???s Next</h3>
+
 #### Commands
 
 curlyq makes use of subcommands, e.g. `curlyq html [options] URL` or `curlyq extract [options] URL`. Each subcommand takes its own options, but I've made an effort to standardize the choices between each command as much as possible.
@@ -440,7 +456,7 @@ COMMAND OPTIONS
 
 Return a hierarchy of all tags in a page. Use `-t` to limit to a specific tag.
 
-    curlyq tags --search '#main .post h3' -q 'attrs[id*=what]' https://brettterpstra.com/2024/01/10/introducing-curlyq-a-pipeline-oriented-curl-helper/
+    curlyq tags --search '#main .post h3' -q '[attrs.id*=what]' https://brettterpstra.com/2024/01/10/introducing-curlyq-a-pipeline-oriented-curl-helper/
 
     [
       {

data/bin/curlyq

@@ -130,13 +130,13 @@ command %i[html curl] do |c|
         out = res.parse(source)
 
         if options[:query]
-          out = out.to_data(url: url, clean: options[:clean]).dot_query(options[:query])
+          out = out.to_data(url: url, clean: options[:clean]).dot_query(options[:query], full_tag: false)
         else
           out = out.to_data
         end
         output.push([out])
       elsif options[:query]
-        queried = res.to_data.dot_query(options[:query])
+        queried = res.to_data.dot_query(options[:query], full_tag: false)
         output.push(queried) if queried
       else
         output.push(res.to_data(url: url))
@@ -356,7 +356,9 @@ command :tags do |c|
       end
     end
 
-    output = output[0] if output.count == 1
+    while output.is_a?(Array) && output.count == 1
+      output = output[0]
+    end
 
     if options[:source]
       puts output.to_html

data/lib/curly/array.rb

@@ -81,8 +81,7 @@ class ::Array
   end
 
   def get_value(path)
-    res = map { |el| el.get_value(path) }
-    res.is_a?(Array) && res.count == 1 ? res[0] : res
+    map { |el| el.get_value(path) }
   end
 
   def to_html

data/lib/curly/hash.rb

@@ -31,9 +31,15 @@ class ::Hash
 
   def get_value(query)
     return nil if self.empty?
+    stringify_keys!
+
     query.split('.').inject(self) do |v, k|
-      k = k.to_i if v.is_a? Array
+      if v.is_a? Array
+        return v.map { |el| el.get_value(k) }
+      end
+      # k = k.to_i if v.is_a? Array
       next unless v.key?(k)
+
       v.fetch(k)
     end
   end
@@ -52,12 +58,18 @@ class ::Hash
       return res.get_value(path)
     end
 
+    path.gsub!(/\[(.*?)\]/) do
+      inter = Regexp.last_match(1).gsub(/\./, '%')
+      "[#{inter}]"
+    end
+
     enumerate = false
     out = []
     q = path.split(/(?<![\d.])\./)
 
     while q.count.positive?
       pth = q.shift
+      pth.gsub!(/%/, '.')
 
       return nil if res.nil?
 
@@ -70,8 +82,8 @@ class ::Hash
 
       ats = []
       at = []
-      while pth =~ /\[[+&,]?\w+( *[\^*$=<>]=? *\w+)?/
-        m = pth.match(/\[(?<com>[,+&])? *(?<key>\w+)( *(?<op>[\^*$=<>]{1,2}) *(?<val>[^,&\]]+))? */)
+      while pth =~ /\[[+&,]?[\w.]+( *[\^*$=<>]=? *\w+)?/
+        m = pth.match(/\[(?<com>[,+&])? *(?<key>[\w.]+)( *(?<op>[\^*$=<>]{1,2}) *(?<val>[^,&\]]+))? */)
 
         comp = [m['key'], m['op'], m['val']]
         case m['com']
@@ -82,7 +94,7 @@ class ::Hash
           at.push(comp)
         end
 
-        pth.sub!(/\[(?<com>[,&+])? *(?<key>\w+)( *(?<op>[\^*$=<>]{1,2}) *(?<val>[^,&\]]+))?/, '[')
+        pth.sub!(/\[(?<com>[,&+])? *(?<key>[\w.]+)( *(?<op>[\^*$=<>]{1,2}) *(?<val>[^,&\]]+))?/, '[')
       end
       ats.push(at) unless at.empty?
       pth.sub!(/\[\]/, '')
@@ -110,11 +122,11 @@ class ::Hash
       pth = ''
 
       return false if res.nil?
+
       if ats.count.positive?
         while ats.count.positive?
           atr = ats.shift
           res = [res] if res.is_a?(Hash)
-
           res.each do |r|
             out.push(full_tag ? tag : r) if evaluate_comp(r, atr)
           end
@@ -140,6 +152,24 @@ class ::Hash
     out
   end
 
+  def array_match(array, key, comp)
+    keep = false
+    array.each do |el|
+      keep = case comp
+             when /^\^/
+               key =~ /^#{el}/i ? true : false
+             when /^\$/
+               key =~ /#{el}$/i ? true : false
+             when /^\*/
+               key =~ /#{el}/i ? true : false
+             else
+               key =~ /^#{el}$/i ? true : false
+             end
+      break if keep
+    end
+    keep
+  end
+
   ##
   ## Evaluate a comparison
   ##
@@ -165,40 +195,57 @@ class ::Hash
             end
       r = r.get_value(key.to_s) if key.to_s =~ /\./
 
-      return r.key?(key) && !r[key].nil? && !r[key].empty? if val.nil?
+      if val.nil?
+        if r.is_a?(Hash)
+          return r.key?(key) && !r[key].nil? && !r[key].empty?
+        elsif r.is_a?(String)
+          return r.nil? ? false : true
+        elsif r.is_a?(Array)
+          return r.empty? ? false : true
+        end
+      end
 
-      if !r.key?(key)
+      if r.nil?
         keep = false
-      elsif r[key].is_a?(Array)
-        valid = r[key].filter do |k|
-          case a[1]
-          when /^\^/
-            k =~ /^#{a[2]}/i ? true : false
-          when /^\$/
-            k =~ /#{a[2]}$/i ? true : false
-          when /^\*/
-            k =~ /#{a[2]}/i ? true : false
+      elsif r.is_a?(Array)
+        valid = r.filter do |k|
+          if k.is_a? Array
+            array_match(k, a[2], a[1])
           else
-            k =~ /^#{a[2]}$/i ? true : false
+            case a[1]
+            when /^\^/
+              k =~ /^#{a[2]}/i ? true : false
+            when /^\$/
+              k =~ /#{a[2]}$/i ? true : false
+            when /^\*/
+              k =~ /#{a[2]}/i ? true : false
+            else
+              k =~ /^#{a[2]}$/i ? true : false
+            end
           end
         end
 
         keep = valid.count.positive?
       elsif val.is_a?(Numeric) && a[1] =~ /^[<>=]{1,2}$/
-        k = r[key].to_i
+        k = r.to_i
         comp = a[1] =~ /^=$/ ? '==' : a[1]
         keep = eval("#{k}#{comp}#{val}")
       else
-        keep = case a[1]
-               when /^\^/
-                 r[key] =~ /^#{a[2]}/i ? true : false
-               when /^\$/
-                 r[key] =~ /#{a[2]}$/i ? true : false
-               when /^\*/
-                 r[key] =~ /#{a[2]}/i ? true : false
-               else
-                 r[key] =~ /^#{a[2]}$/i ? true : false
-               end
+        v = r.is_a?(Hash) ? r[key] : r
+        if v.is_a? Array
+          keep = array_match(v, a[2], a[1])
+        else
+          keep = case a[1]
+                 when /^\^/
+                   v =~ /^#{a[2]}/i ? true : false
+                 when /^\$/
+                   v =~ /#{a[2]}$/i ? true : false
+                 when /^\*/
+                   v =~ /#{a[2]}/i ? true : false
+                 else
+                   v =~ /^#{a[2]}$/i ? true : false
+                 end
+        end
       end
 
       return false unless keep

data/lib/curly/version.rb

@@ -1,3 +1,3 @@
 module Curly
-  VERSION = '0.0.8'
+  VERSION = '0.0.9'
 end

data/src/_README.md

@@ -10,9 +10,12 @@ _If you find this useful, feel free to [buy me some coffee][donate]._
 [donate]: https://brettterpstra.com/donate
 <!--END GITHUB-->
 
+[jq]: https://github.com/jqlang/jq "Command-line JSON processor"
+[yq]: https://github.com/mikefarah/yq "yq is a portable command-line YAML, JSON, XML, CSV, TOML and properties processor"
+
 The current version of `curlyq` is <!--VER-->0.0.4<!--END VER-->.
 
-CurlyQ is a utility that provides a simple interface for curl, with additional features for things like extracting images and links, finding elements by CSS selector or XPath, getting detailed header info, and more. It's designed to be part of a scripting pipeline, outputting everything as structured data (JSON or YAML). It also has rudimentary support for making calls to JSON endpoints easier, but it's expected that you'll use something like `jq` to parse the output.
+CurlyQ is a utility that provides a simple interface for curl, with additional features for things like extracting images and links, finding elements by CSS selector or XPath, getting detailed header info, and more. It's designed to be part of a scripting pipeline, outputting everything as structured data (JSON or YAML). It also has rudimentary support for making calls to JSON endpoints easier, but it's expected that you'll use something like [jq] to parse the output.
 
 [github]: https://github.com/ttscoff/curlyq/
 
@@ -45,6 +48,9 @@ You can shape the results using `--search` (`-s`) and `--query` (`-q`) on some c
 
 A search uses either CSS or XPath syntax to locate elements. For example, if you wanted to locate all of the `<article>` elements with a class of `post` inside of the div with an id of `main`, you would run `--search '#main article.post'`. Searches can target tags, ids, and classes, and can accept `>` to target direct descendents. You can also use XPaths, but I hate those so I'm not going to document them.
 
+> I've tried to make the query function useful, but if you want to do any kind of advanced shaping, you're better off piping the JSON output to [jq] or [yq].
+<!--JEKYLL{:.warn}-->
+
 Queries are specifically for shaping CurlyQ output. If you're using the `html` command, it returns a key called `images`, so you can target just the images in the response with `-q 'images'`. The queries accept array syntax, so to get the first image, you would use `-q 'images[0]'`. Ranges are accepted as well, so `-q 'images[1..4]'` will return the 2nd through 5th images found on the page. You can also do comparisons, e.g. `images[rel=me]'` to target only images with a `rel` attribute of `me`.
 
 The comparisons for the query flag are:
@@ -58,6 +64,16 @@ The comparisons for the query flag are:
 - `^=` starts with text
 - `$=` ends with text
 
+Comparisons can be numeric or string comparisons. A numeric comparison like `curlyq images -q '[width>500]' URL` would return all of the images on the page with a width attribute greater than 500.
+
+You can also use dot syntax inside of comparisons, e.g. `[links.rel*=me]` to target the links object (`html` command), and return only the links with a `rel=me` attribute. If the comparison is to an array object (like `class` or `rel`), it will match if any of the elements of the array match your comparison.
+
+If you end the query with a specific key, only that key will be output. If there's only one match, it will be output as a raw string. If there are multiple matches, output will be an array:
+
+    curlyq tags --search '#main .post h3' -q '[attrs.id*=what].source' 'https://brettterpstra.com/2024/01/10/introducing-curlyq-a-pipeline-oriented-curl-helper/'
+    
+    <h3 id="whats-next">What’s Next</h3>
+
 #### Commands
 
 curlyq makes use of subcommands, e.g. `curlyq html [options] URL` or `curlyq extract [options] URL`. Each subcommand takes its own options, but I've made an effort to standardize the choices between each command as much as possible.
@@ -314,7 +330,7 @@ Example:
 
 Return a hierarchy of all tags in a page. Use `-t` to limit to a specific tag.
 
-    curlyq tags --search '#main .post h3' -q 'attrs[id*=what]' https://brettterpstra.com/2024/01/10/introducing-curlyq-a-pipeline-oriented-curl-helper/
+    curlyq tags --search '#main .post h3' -q '[attrs.id*=what]' https://brettterpstra.com/2024/01/10/introducing-curlyq-a-pipeline-oriented-curl-helper/
 
     [
       {

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: curlyq
 version: !ruby/object:Gem::Version
-  version: 0.0.8
+  version: 0.0.9
 platform: ruby
 authors:
 - Brett Terpstra
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-01-15 00:00:00.000000000 Z
+date: 2024-01-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake