pg_query 2.2.0 → 6.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 27667deffca50f6c8efc514a644d951e5129e5a7ff5fdf472ad39501adda12f0
-  data.tar.gz: 3398f333de65ba7fdc637d9746805525b4a22933cbd9d6889c1d68643a780dca
+  metadata.gz: 6b36a06556f82152c5dbf59f614d17ace5e6b796f47e392e9db0cb8a9b673df2
+  data.tar.gz: db1624f7783f496ecdcba756d4d3d0d2c2d87efbaf2a175d580041c28c1e3e5f
 SHA512:
-  metadata.gz: c30bfcab9a142efc9c0fa8f12a18c157daa2074baf5b792cb832c2a214676f62aa558be69de2d5a0e15f87e3b4818b78d733844ada15fd0e96ba35fbe3b075da
-  data.tar.gz: 1292be5ffdf9021997514e2a8175a4fae4058fbf4ece99c1a49de09b2490d9244947d35b46ecb0574c32a61d4cbb86eee518372b8b20496d47fd86ae264b8489
+  metadata.gz: 0fc23f15402fd98ca1768ea81cb31fbac2bbac488640d4af38bcb20b388ea57f395156ca8bd89192eb89736f721a9e6502e8891c2bd2c0774e38d17034e89ddc
+  data.tar.gz: 385424fe6139048a0fccb5d0ecc1df1f3f4a66aca59032137faad5ce86a5ec43ec28b1db77f43a2b816e9bde1c61fd219a0dd46b88cfd630b9a0b68fef05364f

data/CHANGELOG.md

@@ -4,6 +4,92 @@
 
 * ...
 
+## 6.1.0     2025-04-02
+
+* Upgrade to libpg_query 17-6.1.0
+  - Update to Postgres 17.4, and add recent patches scheduled for Postgres 17.5 (not yet released)
+    - Notably, this pulls in support for macOS 15.4 which defines strchrnul
+    in its standard library, fixing builds on up-to-date macOS versions.
+  - Deparser improvements
+    - Add parenthesis around AT LOCAL / AT TIMEZONE if needed
+    - Correctness improvements related to expressions and function calls
+- Fixes to support newer Rubocop version linting of the codebase
+- Add support for OpenBSD
+
+## 6.0.0     2024-11-26
+
+* Upgrade to libpg_query 17-6.0.0
+  - Updates to the Postgres 17 parser
+  - Deparser improvements:
+    - Add support for deparsing `JSON_TABLE`, `JSON_QUERY`, `JSON_EXISTS`, `JSON_VALUE`
+    - Add support for deparsing `JSON`, `JSON_SCALAR`, `JSON_SERIALIZE`
+    - Add support for deparsing `COPY ... FORCE_NULL(*)`
+    - Add support for deparsing `ALTER COLUMN ... SET EXPRESSION AS`
+    - Add support for deparsing `SET STATISTICS DEFAULT`
+    - Add support for deparsing `SET ACCESS METHOD DEFAULT`
+    - Add support for deparsing `... AT LOCAL`
+    - Add support for deparsing `merge_action()`
+    - Add support for deparsing `MERGE ... RETURNING`
+    - Add support for deparsing `NOT MATCHED [ BY TARGET ]`
+* Improve treewalker performance by avoiding allocations [(#309)](https://github.com/pganalyze/pg_query_go/pull/309)
+* Fix FreeBSD builds [(#313)](https://github.com/pganalyze/pg_query_go/pull/313)
+
+## 5.1.0     2024-01-09
+
+* Update to libpg_query 16-5.1.0
+  - Add support for running on Windows
+  - Add support for compiling on 32-bit systems
+* Treewalker: Allow passing a block with a single argument to walk!
+* PgQuery::Node: Add inner and inner= helpers to get/set inner object
+
+
+## 5.0.0     2023-12-22
+
+* Upgrade to libpg_query 16-5.0.0
+  - Updates to the Postgres 16 parser
+  - Multiple deparser improvements
+
+
+## 4.2.3     2023-08-04
+
+* Update to libpg_query 15-4.2.3
+  - Fix builds when compiling with `glibc >=  2.38` [#203](https://github.com/pganalyze/libpg_query/pull/203)
+  - Deparser: Add support for COALESCE and other expressions in LIMIT clause [#199](https://github.com/pganalyze/libpg_query/pull/199)
+
+## 4.2.2     2023-07-07
+
+* Update to libpg_query 15-4.2.2
+  - Deparser: Add support for multi-statement CREATE PROCEDURE definitions
+  - Deparser: Correctly quote identifier in ALTER TABLE ... ADD CONSTRAINT [x]
+  - Deparser: Add support for index fillfactor within CREATE TABLE, fix SHOW ALL
+* Fix builds on FreeBSD ([#292](https://github.com/pganalyze/pg_query/pull/292))
+  - This was broken since 4.2.0, due to pg_query_ruby_freebsd.sym being removed by accident
+
+
+## 4.2.1     2023-05-19
+
+* Parse: Fix `ALTER INDEX my_index_name` to return `tables=[]` ([#285](https://github.com/pganalyze/pg_query/pull/285))
+* Parse: Detect tables in a SELECT INTO clause as DDL tables ([#281](https://github.com/pganalyze/pg_query/pull/281))
+* Add support for Ruby 3.2 ([#283](https://github.com/pganalyze/pg_query/pull/283))
+* Bump up `google-protobuf` dependency to `>= 3.22.3`
+  - 3.22.0 or newer is required for Ruby 3.2 support
+* Update to libpg_query 15-4.2.1
+  - Deparser: Handle INTERVAL correctly when used in SET statements
+  - Deparser: Ensure index names are quoted as identifiers
+
+
+## 4.2.0     2023-02-08
+
+* Update to libpg_query 15-4.2.0
+  - Update to PostgreSQL 15.1
+
+
+## 2.2.1     2023-01-20
+
+* Detect tables used in the query of a PREPARE statement ([#273](https://github.com/pganalyze/pg_query/pull/273))
+* Expose recursive walk functionality via walk! ([#268](https://github.com/pganalyze/pg_query/pull/268))
+* Retain schema in name when parsing out functions ([#272](https://github.com/pganalyze/pg_query/pull/272))
+
 
 ## 2.2.0     2022-11-02
 

data/README.md

@@ -2,11 +2,11 @@
 
 This Ruby extension uses the actual PostgreSQL server source to parse SQL queries and return the internal PostgreSQL parsetree.
 
-In addition the extension allows you to normalize queries (replacing constant values with ?) and parse these normalized queries into a parsetree again.
+In addition the extension allows you to normalize queries (replacing constant values with $n) and parse these normalized queries into a parsetree again.
 
 When you build this extension, it builds parts of the PostgreSQL server source (see [libpg_query](https://github.com/pganalyze/libpg_query)), and then statically links it into this extension.
 
-This is slightly crazy, but is the only reliable way of parsing all valid PostgreSQL queries.
+This may seem like a lot of complexity, but is the only reliable way of parsing all valid PostgreSQL queries.
 
 You can find further examples and a longer rationale here: https://pganalyze.com/blog/parse-postgresql-queries-in-ruby.html
 
@@ -25,10 +25,10 @@ Due to compiling parts of PostgreSQL, installation might take a while on slower
 ```ruby
 PgQuery.parse("SELECT 1")
 
-=> #<PgQuery::ParserResult:0x00007fb69a958820
+=> #<PgQuery::ParserResult:0x000000012ec4e9e0
   @query="SELECT 1",
   @tree=<PgQuery::ParseResult:
-    version: 130002,
+    version: 160001,
     stmts: [
       <PgQuery::RawStmt:
         stmt: <PgQuery::Node:
@@ -41,9 +41,8 @@ PgQuery.parse("SELECT 1")
                   indirection: [],
                   val: <PgQuery::Node:
                     a_const: <PgQuery::A_Const:
-                      val: <PgQuery::Node:
-                        integer: <PgQuery::Integer: ival: 1>
-                      >,
+                      ival: <PgQuery::Integer: ival: 1>,
+                      isnull: false,
                       location: 7
                     >
                   >,
@@ -53,6 +52,7 @@ PgQuery.parse("SELECT 1")
             ],
             from_clause: [],
             group_clause: [],
+            group_distinct: false,
             window_clause: [],
             values_lists: [],
             sort_clause: [],
@@ -67,11 +67,18 @@ PgQuery.parse("SELECT 1")
       >
     ]
   >,
-  @warnings=[]>
+  @warnings=[],
+  @tables=nil,
+  @aliases=nil,
+  @cte_names=nil,
+  @functions=nil
+>
 ```
 
 ### Modifying a parsed query and turning it into SQL again
 
+This is a simple example for `deparse`, for more complex modification, use `walk!`.
+
 ```ruby
 parsed_query = PgQuery.parse("SELECT * FROM users")
 
@@ -90,20 +97,12 @@ parsed_query.deparse
 PgQuery.normalize("SELECT 1 FROM x WHERE y = 'foo'")
 
 => "SELECT $1 FROM x WHERE y = $2"
-
-# Parsing a normalized query (pre-Postgres 10 style)
-PgQuery.parse("SELECT ? FROM x WHERE y = ?")
-
-=> #<PgQuery::ParserResult:0x00007fb69a97a5d8
-  @query="SELECT ? FROM x WHERE y = ?",
-  @tree=<PgQuery::ParseResult: ...>,
-  @warnings=[]>
 ```
 
 ### Extracting tables from a query
 
 ```ruby
-PgQuery.parse("SELECT ? FROM x JOIN y USING (id) WHERE z = ?").tables
+PgQuery.parse("SELECT $1 FROM x JOIN y USING (id) WHERE z = $2").tables
 
 => ["x", "y"]
 ```
@@ -111,7 +110,7 @@ PgQuery.parse("SELECT ? FROM x JOIN y USING (id) WHERE z = ?").tables
 ### Extracting columns from a query
 
 ```ruby
-PgQuery.parse("SELECT ? FROM x WHERE x.y = ? AND z = ?").filter_columns
+PgQuery.parse("SELECT $1 FROM x WHERE x.y = $2 AND z = $3").filter_columns
 
 => [["x", "y"], [nil, "z"]]
 ```
@@ -128,7 +127,7 @@ PgQuery.parse("SELECT 2; --- comment").fingerprint
 => "50fde20626009aba"
 
 # Faster fingerprint method that is implemented inside the native C library
-PgQuery.fingerprint("SELECT ?")
+PgQuery.fingerprint("SELECT $1")
 
 => "50fde20626009aba"
 ```
@@ -138,27 +137,52 @@ PgQuery.fingerprint("SELECT ?")
 ```ruby
 PgQuery.scan('SELECT 1 --comment')
 
-=> [<PgQuery::ScanResult: version: 130002, tokens: [
+=> [<PgQuery::ScanResult: version: 160001, tokens: [
 <PgQuery::ScanToken: start: 0, end: 6, token: :SELECT, keyword_kind: :RESERVED_KEYWORD>,
 <PgQuery::ScanToken: start: 7, end: 8, token: :ICONST, keyword_kind: :NO_KEYWORD>,
 <PgQuery::ScanToken: start: 9, end: 18, token: :SQL_COMMENT, keyword_kind: :NO_KEYWORD>]>,
  []]
 ```
 
-## Differences from Upstream PostgreSQL
+### Walking the parse tree
+
+For generalized use, PgQuery provides `walk!` as a means to recursively work with the parsed query.
+
+This can be used to create a bespoke pretty printer:
+
+```ruby
+parsed_query = PgQuery.parse "SELECT * FROM tbl"
+parsed_query.walk! { |node, k, v, location| puts k }
+```
+
+More usefully, this can be used to rewrite a query. For example:
+
+```ruby
+parsed_query.walk! do |node, k, v, location|
+  next unless k.eql?(:range_var) || k.eql?(:relation)
+  next if v.relname.nil?
+  v.relname = "X_" + v.relname
+end
+
+parsed_query.deparse
+```
+
+There are some caveats, and limitations, in this example.
+
+First, some of the tree nodes are frozen. You can replace them, but you cannot modify in place.
 
-This gem is based on [libpg_query](https://github.com/pganalyze/libpg_query),
-which uses the latest stable PostgreSQL version, but with a patch applied
-to support parsing normalized queries containing `?` replacement characters.
+Second, table rewriting is a bit more nuanced than this example. While this will rewrite the table names, it will
+not correctly handle all CTEs, or rewrite columns with explicit table names.
 
 ## Supported Ruby Versions
 
 Currently tested and officially supported Ruby versions:
 
-* CRuby 2.5
-* CRuby 2.6
 * CRuby 2.7
 * CRuby 3.0
+* CRuby 3.1
+* CRuby 3.2
+* CRuby 3.3
 
 Not supported:
 
@@ -182,7 +206,7 @@ Once that is done, follow the following steps:
 
 ## Resources
 
-See [libpg_query](https://github.com/pganalyze/libpg_query/blob/13-latest/README.md#resources) for pg_query in other languages, as well as products/tools built on pg_query.
+See [libpg_query](https://github.com/pganalyze/libpg_query/blob/15-latest/README.md#resources) for pg_query in other languages, as well as products/tools built on pg_query.
 
 ## Original Author
 
@@ -196,8 +220,10 @@ See [libpg_query](https://github.com/pganalyze/libpg_query/blob/13-latest/README
 
 ## License
 
-Copyright (c) 2015, pganalyze Team <team@pganalyze.com><br>
-pg_query is licensed under the 3-clause BSD license, see LICENSE file for details.
+PostgreSQL server source code, used under the [PostgreSQL license](https://www.postgresql.org/about/licence/).<br>
+Portions Copyright (c) 1996-2023, The PostgreSQL Global Development Group<br>
+Portions Copyright (c) 1994, The Regents of the University of California
 
-Query normalization code:<br>
-Copyright (c) 2008-2015, PostgreSQL Global Development Group
+All other parts are licensed under the 3-clause BSD license, see LICENSE file for details.<br>
+Copyright (c) 2015, Lukas Fittl <lukas@fittl.com><br>
+Copyright (c) 2016-2023, Duboce Labs, Inc. (pganalyze) <team@pganalyze.com>

data/Rakefile

@@ -5,8 +5,8 @@ require 'rspec/core/rake_task'
 require 'rubocop/rake_task'
 require 'open-uri'
 
-LIB_PG_QUERY_TAG = '13-2.2.0'.freeze
-LIB_PG_QUERY_SHA256SUM = '07916be1a2b780dee6feed936aaa04ccee2a3afde8570a6920c3a839c87539c6'.freeze
+LIB_PG_QUERY_TAG = '17-6.1.0'.freeze
+LIB_PG_QUERY_SHA256SUM = 'a3dc0e4084a23da35128d4e9809ff27241c29a44fde74ba40a378b33d2cdefe2'.freeze
 
 Rake::ExtensionTask.new 'pg_query' do |ext|
   ext.lib_dir = 'lib/pg_query'
@@ -55,18 +55,17 @@ task :update_source do
   # Backup important files from ext dir
   system("rm -fr #{extbakdir}")
   system("mkdir -p #{extbakdir}")
-  system("cp -a #{extdir}/pg_query_ruby.{c,sym} #{extdir}/extconf.rb #{extbakdir}")
+  system("cp -a #{extdir}/pg_query_ruby.c #{extdir}/ext_symbols*.sym #{extdir}/extconf.rb #{extbakdir}")
 
   FileUtils.rm_rf extdir
 
   # Reduce everything down to one directory
   system("mkdir -p #{extdir}")
   system("cp -a #{libdir}/src/* #{extdir}/")
+  system("mv #{extdir}/postgres/include #{extdir}/include/postgres")
   system("mv #{extdir}/postgres/* #{extdir}/")
   system("rmdir #{extdir}/postgres")
   system("cp -a #{libdir}/pg_query.h #{extdir}/include")
-  # Make sure every .c file in the top-level directory is its own translation unit
-  system("mv #{extdir}/*{_conds,_defs,_helper}.c #{extdir}/include")
   # Protobuf definitions
   system("protoc --proto_path=#{libdir}/protobuf --ruby_out=#{File.join(__dir__, 'lib/pg_query')} #{libdir}/protobuf/pg_query.proto")
   system("mkdir -p #{extdir}/include/protobuf")
@@ -85,5 +84,5 @@ task :update_source do
   # Other support files
   system("cp -a #{libdir}/testdata/* #{testfilesdir}")
   # Copy back the custom ext files
-  system("cp -a #{extbakdir}/pg_query_ruby.{c,sym} #{extbakdir}/extconf.rb #{extdir}")
+  system("cp -a #{extbakdir}/pg_query_ruby.c #{extbakdir}/ext_symbols*.sym #{extbakdir}/extconf.rb #{extdir}")
 end

data/ext/pg_query/ext_symbols_freebsd.sym

@@ -0,0 +1 @@
+Init_pg_query

data/ext/pg_query/ext_symbols_freebsd_with_ruby_abi_version.sym

@@ -0,0 +1,2 @@
+Init_pg_query
+ruby_abi_version

data/ext/pg_query/ext_symbols_openbsd.sym

@@ -0,0 +1 @@
+Init_pg_query

data/ext/pg_query/ext_symbols_openbsd_with_ruby_abi_version.sym

@@ -0,0 +1,2 @@
+Init_pg_query
+ruby_abi_version

data/ext/pg_query/ext_symbols_with_ruby_abi_version.sym

@@ -0,0 +1,2 @@
+_Init_pg_query
+_ruby_abi_version

data/ext/pg_query/extconf.rb

@@ -7,20 +7,44 @@ require 'pathname'
 
 $objs = Dir.glob(File.join(__dir__, '*.c')).map { |f| Pathname.new(f).sub_ext('.o').to_s }
 
-$CFLAGS << " -fvisibility=hidden -O3 -Wall -fno-strict-aliasing -fwrapv -fstack-protector -Wno-unused-function -Wno-unused-variable -Wno-clobbered -Wno-sign-compare -Wno-discarded-qualifiers -g"
+if RUBY_PLATFORM !~ /cygwin|mswin|mingw|bccwin|wince|emx/
+  $CFLAGS << " -fvisibility=hidden -O3 -Wall -fno-strict-aliasing -fwrapv -fstack-protector -Wno-unused-function -Wno-unused-variable -Wno-clobbered -Wno-sign-compare -Wno-discarded-qualifiers -Wno-unknown-warning-option -g"
+end
+
+$INCFLAGS = "-I#{File.join(__dir__, 'include')} " + "-I#{File.join(__dir__, 'include', 'postgres')} " + $INCFLAGS
 
-$INCFLAGS = "-I#{File.join(__dir__, 'include')} " + $INCFLAGS
+if RUBY_PLATFORM =~ /cygwin|mswin|mingw|bccwin|wince|emx/
+  $INCFLAGS = "-I#{File.join(__dir__, 'include', 'postgres', 'port', 'win32')} " + $INCFLAGS
+end
+
+if RUBY_PLATFORM =~ /mswin/
+  $INCFLAGS = "-I#{File.join(__dir__, 'include', 'postgres', 'port', 'win32_msvc')} " + $INCFLAGS
+end
+
+# "ruby_abi_version" is a required symbol to be exported on Ruby 3.2+ development releases
+# See https://github.com/ruby/ruby/pull/5474 and https://github.com/ruby/ruby/pull/6231
+def export_ruby_abi_version
+  return false if RUBY_PATCHLEVEL >= 0 # Not a development release
+  m = /(\d+)\.(\d+)/.match(RUBY_VERSION)
+  return false if m.nil?
+  major = m[1].to_i
+  minor = m[2].to_i
+  major >= 3 && minor >= 2
+end
+
+def ext_symbols_filename
+  name = 'ext_symbols'
+  name += '_freebsd' if RUBY_PLATFORM =~ /freebsd/
+  name += '_openbsd' if RUBY_PLATFORM =~ /openbsd/
+  name += '_with_ruby_abi_version' if export_ruby_abi_version
+  "#{name}.sym"
+end
 
-SYMFILE =
-  if RUBY_PLATFORM =~ /freebsd/
-    File.join(__dir__, 'pg_query_ruby_freebsd.sym')
-  else
-    File.join(__dir__, 'pg_query_ruby.sym')
-  end
+SYMFILE = File.join(__dir__, ext_symbols_filename)
 
 if RUBY_PLATFORM =~ /darwin/
   $DLDFLAGS << " -Wl,-exported_symbols_list #{SYMFILE}" unless defined?(::Rubinius)
-else
+elsif RUBY_PLATFORM !~ /cygwin|mswin|mingw|bccwin|wince|emx/
   $DLDFLAGS << " -Wl,--retain-symbols-file=#{SYMFILE}"
 end
 

data/ext/pg_query/include/pg_query.h

@@ -2,6 +2,7 @@
 #define PG_QUERY_H
 
 #include <stdint.h>
+#include <sys/types.h>
 
 typedef struct {
 	char* message; // exception message
@@ -13,7 +14,7 @@ typedef struct {
 } PgQueryError;
 
 typedef struct {
-  unsigned int len;
+  size_t len;
   char* data;
 } PgQueryProtobuf;
 
@@ -69,17 +70,42 @@ typedef struct {
   PgQueryError* error;
 } PgQueryNormalizeResult;
 
+// Postgres parser options (parse mode and GUCs that affect parsing)
+
+typedef enum
+{
+	PG_QUERY_PARSE_DEFAULT = 0,
+	PG_QUERY_PARSE_TYPE_NAME,
+	PG_QUERY_PARSE_PLPGSQL_EXPR,
+	PG_QUERY_PARSE_PLPGSQL_ASSIGN1,
+	PG_QUERY_PARSE_PLPGSQL_ASSIGN2,
+	PG_QUERY_PARSE_PLPGSQL_ASSIGN3
+} PgQueryParseMode;
+
+// We technically only need 3 bits to store parse mode, but
+// having 4 bits avoids API breaks if another one gets added.
+#define PG_QUERY_PARSE_MODE_BITS 4
+#define PG_QUERY_PARSE_MODE_BITMASK ((1 << PG_QUERY_PARSE_MODE_BITS) - 1)
+
+#define PG_QUERY_DISABLE_BACKSLASH_QUOTE 16 // backslash_quote = off (default is safe_encoding, which is effectively on)
+#define PG_QUERY_DISABLE_STANDARD_CONFORMING_STRINGS 32 // standard_conforming_strings = off (default is on)
+#define PG_QUERY_DISABLE_ESCAPE_STRING_WARNING 64 // escape_string_warning = off (default is on)
+
 #ifdef __cplusplus
 extern "C" {
 #endif
 
 PgQueryNormalizeResult pg_query_normalize(const char* input);
+PgQueryNormalizeResult pg_query_normalize_utility(const char* input);
 PgQueryScanResult pg_query_scan(const char* input);
 PgQueryParseResult pg_query_parse(const char* input);
+PgQueryParseResult pg_query_parse_opts(const char* input, int parser_options);
 PgQueryProtobufParseResult pg_query_parse_protobuf(const char* input);
+PgQueryProtobufParseResult pg_query_parse_protobuf_opts(const char* input, int parser_options);
 PgQueryPlpgsqlParseResult pg_query_parse_plpgsql(const char* input);
 
 PgQueryFingerprintResult pg_query_fingerprint(const char* input);
+PgQueryFingerprintResult pg_query_fingerprint_opts(const char* input, int parser_options);
 
 // Use pg_query_split_with_scanner when you need to split statements that may
 // contain parse errors, otherwise pg_query_split_with_parser is recommended
@@ -106,9 +132,9 @@ void pg_query_free_fingerprint_result(PgQueryFingerprintResult result);
 void pg_query_exit(void);
 
 // Postgres version information
-#define PG_VERSION "13.8"
-#define PG_MAJORVERSION "13"
-#define PG_VERSION_NUM 130008
+#define PG_MAJORVERSION "17"
+#define PG_VERSION "17.4"
+#define PG_VERSION_NUM 170004
 
 // Deprecated APIs below