prestogres 0.3.0 → 0.4.0

data/ChangeLog

@@ -1,4 +1,15 @@
 
+2014-03-05 version 0.4.0:
+
+* Preserve OID of tables when clients SELECT from system catalogs
+* Increased maximum rewritten query length to 32768 bytes
+* Fixed protocol version v2 handling
+* Fixed 'too long error message' errors
+* Fixed build scripts
+* Add "LIMIT 1000" to "SELECT * FROM table" queries automatically
+  (ad-hoc fix for Tableau)
+
+
 2014-02-07 version 0.3.0:
 
 * Fixed system catalog queries to support multiple schemas

data/README.md

@@ -14,54 +14,92 @@ You can use any PostgreSQL clients (see also *Limitation* section):
 
 Prestogres also offers password-based authentication and SSL.
 
+## Documents
+
+* [How it works?](#how-it-works)
+* [Limitation](#limitation)
+* [Installation](#installation)
+* [Running servers](#running-servers)
+  * [Mac OS X](#mac-os-x)
+* [Configuration](#configuration)
+  * [pgpool.conf file](#pgpoolconf-file)
+  * [pool_hba.conf file](#pool_hbaconf-file)
+     * [prestogres_md5 method](#prestogres_md5-method)
+     * [prestogres_external method](#prestogres_external-method)
+  * [prestogres command](#prestogres-command)
+* [Development](#development)
+
+---
+
 ## How it works?
 
 Prestogres uses modified **[pgpool-II](http://www.pgpool.net/)** to rewrite queries before sending them to PostgreSQL.
-pgpool-II is originally an open-source middleware to provide connection pool and other features to PostgreSQL.
+pgpool-II is originally an open-source middleware to provide connection pool and other features to PostgreSQL. For regular SQL query, patched pgpool-II wraps it in **run_presto_astemp_table(..., 'SELECT ... FROM ...')** function. This function sends the query to Presto:
 
-```
-       PostgreSQL protocol                     Presto protocol (HTTP)
-            /                                      /
-           /  +-----------+      +------------+   /  +--------+
-  client ---> | pgpool-II | ---> | PostgreSQL | ---> | Presto |
-              +-----------+      +------------+      +--------+
-                       \                   \
-                    rewrite queries       run custom functions
-            |                                   |
-            +-----------------------------------+
-                         Prestogres
-```
+![Regular SQL](https://gist.github.com/frsyuki/9012980/raw/2782f51cd3c708254ac64c4c30b7bff0e7894640/figure1.png)
+
+If the query selects records from system catalog (e.g. `\\d` command by psql), patched pgpool-II wraps the query in `run_system_catalog_as_temp_table` function. It gets table list from Presto, and actually runs **CREATE TABLE** to create records in system catalog on PostgreSQL. Then the original query runs as usual on PostgreSQL:
+
+![System catalog access](https://gist.github.com/frsyuki/9012980/raw/f869bd3dd59ab22f439165972c29d122ea560694/figure2.png)
 
-1. **Client** sends a query to a patched pgpool-II
-2. **pgpool-II** rewrites the query to `SELECT run_presto_as_temp_table(..., '...original SELECT query...')` + `SELECT * FROM presto_result`.
-3. **PostgreSQL** runs `run_presto_as_temp_table`. This custom function runs a query on Presto and writes results into a temporary table `presto_result`.
+In fact there're some other tricks. See [pgsql/prestogres.py](pgsql/prestogres.py) for the real behavior.
+
+---
 
 ## Limitation
 
 * Extended query is not supported ([PostgreSQL Frontend/Backend Protocol](http://www.postgresql.org/docs/9.3/static/protocol.html))
-  * ODBC driver needs to set **UseServerSidePrepare=0** (Server side prepare: no) property
-  * ODBC driver needs to use "Unicode" mode
-  * JDBC driver needs to set **protocolVersion=2** property
-* DECLARE/FETCH is not supported
+  * ODBC driver needs to set:
+     * **UseServerSidePrepare=0** (Server side prepare: no) property
+     * **UseDeclareFetch=0** (Use Declare/Fetch: no) property
+     * **Unicode** mode
+  * JDBC driver needs to set:
+     * **protocolVersion=2** property
+* Cursor (DECLARE/FETCH) is not supported
+
+---
 
 ## Installation
 
+### 1. Install PostgreSQL >= 9.3
+
+You need to install PostgreSQL separately. Following commands install PostgreSQL 9.3 from postgresql.org:
+
+**Ubuntu/Debian:**
+
 ```
-$ gem install prestogres --no-ri --no-rdoc
+wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
+sudo apt-get update
+sudo apt-get install postgresql-9.3 postgresql-server-dev-9.3 postgresql-plpython-9.3
+sudo apt-get install gcc make libssl-dev libpcre3-dev
+sudo apt-get install ruby ruby-dev
 ```
 
-Prestogres package installs patched pgpool-II but doesn't install PostgreSQL. You need to install PostgreSQL >= 9.3 (with python support) separately.
+**RedHat/CentOS:**
 
-* If you don't have **gem** command, install Ruby >= 1.9.0 first
-* If installation failed, you may need to install following packages using apt or yum:
-  * basic toolchain (gcc, make, etc.)
-  * OpenSSL dev package (Debian/Ubuntu: **libssl-dev**, RedHat/CentOS: **openssl-dev**)
-  * PostgreSQL server dev package (Debian/Ubuntu: **postgresql-server-dev-9.3**, RedHat/CentOS: **postgresql-devel**)
+```
+sudo yum install http://yum.postgresql.org/9.3/redhat/rhel-6-x86_64/pgdg-redhat93-9.3-1.noarch.rpm
+sudo yum install postgresql93-server postgresql93-contrib postgresql93-devel
+sudo yum install gcc make openssl-devel pcre-devel
+sudo yum install ruby ruby-devel
+```
+
+**Mac OS X:**
 
-### Installation FAQ
+```
+ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
+brew install postgresql
+```
+
+### 2. Install Prestogres
+
+```
+$ sudo gem install prestogres --no-ri --no-rdoc
+```
 
+---
 
-## Runing servers
+## Running servers
 
 You need to run 2 server programs: pgpool-II and PostgreSQL.
 You can use `prestogres` command to setup & run them as following:
@@ -86,26 +124,42 @@ $ psql -h localhost -p 9900 -U pg postgres
 
 If configuration is correct, you can run `SELECT * FROM sys.node;` query. Otherwise, see log files in **./pgdata/log/** directory.
 
-#### Mac OS X
+#### Setting shmem max parameter
 
-You need to run following commands before running PostgreSQL on Mac OS X:
+Probably above command fails first time! Error message is:
 
 ```
-$ sudo sysctl -w kern.sysv.shmmax=1073741824
-$ sudo sysctl -w kern.sysv.shmall=1073741824
+FATAL:  could not create shared memory segment: Cannot allocate memory
+DETAIL:  Failed system call was shmget(key=6432001, size=3809280, 03600).
+HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded
+available memory or swap space, or exceeded your kernel's SHMALL parameter.  You can either
+reduce the request size or reconfigure the kernel with larger SHMALL.  To reduce the request
+size (currently 3809280 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing
+shared_buffers or max_connections.
 ```
 
-if you got following log messages:
+You need to set 2 kernel parameters to run PostgreSQL.
+
+**Linux:**
 
 ```
-FATAL:  could not create shared memory segment: Cannot allocate memory
-DETAIL:  Failed system call was shmget(key=6432001, size=3809280, 03600).
-HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space, or exceeded your kernel's SHMALL parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMALL.  To reduce the request size (currently 3809280 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
+sudo sudo -c "echo kernel.shmmax = 17179869184 >> /etc/sysctl.conf"
+sudo sudo -c "echo kernel.shmall = 4194304 >> /etc/sysctl.conf"
+sudo sysctl -p /etc/sysctl.conf
 ```
 
+**Mac OS X:**
+
+```
+$ sudo sysctl -w kern.sysv.shmmax=1073741824
+$ sudo sysctl -w kern.sysv.shmall=1073741824
+```
+
+---
+
 ## Configuration
 
-### pgool.conf file
+### pgpool.conf file
 
 Please read [pgpool-II documentation](http://www.pgpool.net/docs/latest/pgpool-en.html) for most of parameters.
 Following parameters are unique to Prestogres:
@@ -184,7 +238,6 @@ See *pgool.conf file* section for available parameters.
 
 If you want to reject this connection, the program exists with non-0 status code.
 
-
 ### Creating database on PostgreSQL
 
 Prestogres setups a database named *postgres* on PostgreSQL by default. But you may want to create other databases
@@ -221,6 +274,8 @@ commands:
   show_init_sql         display statements to initialize a new PostgreSQL database
 ```
 
+---
+
 ## Development
 
 To install git HEAD, use following commands to build:
@@ -242,3 +297,9 @@ $ bundle exec rake
 $ gem install --no-ri --no-rdoc pkg/prestogres-*.gem
 # if this command failed, you may need to install toolchain (gcc, etc.) to build pgpool-II
 ```
+
+___
+
+    Prestogres is licensed under Apache License, Version 2.0.
+    Copyright (C) 2014 Sadayuki Furuhashi
+

data/VERSION

@@ -1 +1 @@
-0.3.0
+0.4.0

data/bin/prestogres

@@ -68,11 +68,11 @@ usage nil unless config[:command]
 args = ARGV[arg_offset..-1]
 
 setup_params = {
- port: 9900,
- backend_port: 6432,
- pgpool_pid_file: nil,
- pgpool_status_dir: nil,
- unix_socket_directory: nil,
+  port: 9900,
+  backend_port: 6432,
+  pgpool_pid_file: nil,
+  pgpool_status_dir: nil,
+  unix_socket_directory: nil,
 }
 
 ENV['PATH'] = "#{ENV['PATH']}:#{config[:bin_path]}"

data/config/pgpool.conf

@@ -269,7 +269,7 @@ white_function_list = ''
                                    # Comma separated list of function names
                                    # that don't write to database
                                    # Regexp are accepted
-black_function_list = 'currval,lastval,nextval,setval,current_setting'
+black_function_list = 'currval,lastval,nextval,setval,current_setting,version,pg_client_encoding'
                                    # Comma separated list of function names
                                    # that write to database
                                    # Regexp are accepted

data/ext/depend

@@ -6,29 +6,28 @@ narchdir = $(if $(rubyarchdir),$(rubyarchdir),$(RUBYARCHDIR))
 # hack for RubyGems >= 1.8.26
 final_archdir = $(if $(findstring .gem.,$(narchdir)),$(dir $(narchdir)),$(narchdir))
 
-INSTALL_DATA = $(realpath $(final_archdir))/prestogres
+NATIVE_PREFIX = $(abspath $(final_archdir)/prestogres)
 
-PGPOOL2_PATH = $(realpath $(srcdir)/../pgpool2)
+CFLAGS += -DPRESTOGRES_PREFIX="\"$(NATIVE_PREFIX)\""
 
-CFLAGS += -DPRESTOGRES_PREFIX="\"$(INSTALL_DATA)\""
+all: native
+clean: clean-native
+install: install-native
 
-all: pgpool2
+PGPOOL2_PATH = $(abspath $(srcdir)/../pgpool2)
 
-clean: clean-pgpool2
-
-install: install-pgpool2
-
-pgpool2:
+native:
 	mkdir -p build
 	cd build && "$(PGPOOL2_PATH)/configure" \
 		CFLAGS="-I$(PGPOOL2_PATH)" \
 		--with-openssl \
-		--prefix="$(INSTALL_DATA)"
+		--prefix="$(NATIVE_PREFIX)"
 	cd build && make sysconfdir=/opt/prestogres/etc
 
-clean-pgpool2:
+clean-native:
 	@rm -rf build
 
-install-pgpool2:
+install-native:
 	cd build && make install
 
+.PHONY: native clean-native install-native

data/pgpool2/child.c

@@ -99,7 +99,7 @@ static void rebuild_startup_packet(StartupPacket *sp)
 {
 	if (sp->major == PROTO_MAJOR_V2) {
 		StartupPacket_v2 *sp2 = (StartupPacket_v2 *)(sp->startup_packet);
-		strncpy(sp2->database, sp2->database, SM_DATABASE);
+		strncpy(sp2->database, sp->database, SM_DATABASE);
 		strncpy(sp2->user, sp->user, SM_USER);
 
 	} else if (sp->major == PROTO_MAJOR_V3) {

data/pgpool2/config.h.in

@@ -79,6 +79,9 @@
 /* Define to 1 if you have the `pam' library (-lpam). */
 #undef HAVE_LIBPAM
 
+/* Define to 1 if you have the `pcre' library (-lpcre). */
+#undef HAVE_LIBPCRE
+
 /* Define to 1 if you have the `pq' library (-lpq). */
 #undef HAVE_LIBPQ
 

data/pgpool2/configure

@@ -11450,6 +11450,45 @@ _ACEOF
 
 fi
 
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for main in -lpcre" >&5
+$as_echo_n "checking for main in -lpcre... " >&6; }
+if test "${ac_cv_lib_pcre_main+set}" = set; then :
+  $as_echo_n "(cached) " >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lpcre  $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+
+int
+main ()
+{
+return main ();
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+  ac_cv_lib_pcre_main=yes
+else
+  ac_cv_lib_pcre_main=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pcre_main" >&5
+$as_echo "$ac_cv_lib_pcre_main" >&6; }
+if test "x$ac_cv_lib_pcre_main" = x""yes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBPCRE 1
+_ACEOF
+
+  LIBS="-lpcre $LIBS"
+
+fi
+
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ANSI C header files" >&5
 $as_echo_n "checking for ANSI C header files... " >&6; }

data/pgpool2/configure.in

@@ -63,6 +63,7 @@ AC_CHECK_LIB(gen,      main)
 AC_CHECK_LIB(PW,       main)
 AC_CHECK_LIB(resolv,   main)
 AC_CHECK_LIB(crypt,   main)
+AC_CHECK_LIB(pcre,     main)
 
 dnl Checks for header files.
 AC_HEADER_STDC

data/pgpool2/pool_query_context.c

@@ -32,6 +32,8 @@
 #include <netinet/in.h>
 #include <stdlib.h>
 
+#include <pcre.h>
+
 /*
  * Where to send query
  */
@@ -293,7 +295,11 @@ int pool_virtual_master_db_node_id(void)
 #define PRESTO_RESULT_TABLE_NAME "presto_result"
 #endif
 
-char rewrite_query_string_buffer[QUERY_STRING_BUFFER_LEN];
+#ifndef PRESTO_REWRITE_QUERY_SIZE_LIMIT
+#define PRESTO_REWRITE_QUERY_SIZE_LIMIT 32768
+#endif
+
+char rewrite_query_string_buffer[PRESTO_REWRITE_QUERY_SIZE_LIMIT];
 
 static char *strcpy_capped(char *buffer, int length, const char *string)
 {
@@ -362,7 +368,7 @@ static void do_replace_query(POOL_QUERY_CONTEXT* query_context, const char *quer
 	query_context->original_length = strlen(dupq) + 1;
 }
 
-static void rewrite_error_query(POOL_QUERY_CONTEXT* query_context, const char *message, const char* errcode)
+static void rewrite_error_query_static(POOL_QUERY_CONTEXT* query_context, const char *message, const char* errcode)
 {
 	char *buffer, *bufend;
 
@@ -381,14 +387,39 @@ static void rewrite_error_query(POOL_QUERY_CONTEXT* query_context, const char *m
 	buffer = strcpy_capped(buffer, bufend - buffer, "; end $$ language plpgsql");
 
 	if (buffer == NULL) {
-		do_replace_query(query_context,
-				"do $$ begin raise ecxeption 'too long error message'; end $$ language plpgsql");
-		return;
+		buffer = rewrite_query_string_buffer;
+		bufend = buffer + sizeof(rewrite_query_string_buffer);
+
+		buffer = strcpy_capped(buffer, bufend - buffer, "do $$ begin raise ecxeption 'too long error message'");
+		buffer = strcpy_capped(buffer, bufend - buffer, " using errcode = E'");
+		buffer = strcpy_capped_escaped(buffer, bufend - buffer, errcode, "'\\$");
+		buffer = strcpy_capped(buffer, bufend - buffer, "'; end $$ language plpgsql");
+
+		if (buffer == NULL) {
+			do_replace_query(query_context,
+					"do $$ begin raise ecxeption 'too long error message'; end $$ language plpgsql");
+			return;
+		}
 	}
 
 	do_replace_query(query_context, rewrite_query_string_buffer);
 }
 
+static void rewrite_error_query(POOL_QUERY_CONTEXT* query_context, char *message, const char* errcode)
+{
+	/* 20 is for escape characters */
+	const size_t static_length = strlen("do $$ begin raise exception '%', E'' using errcode = E'XXXXX'; end $$ language plpgsql") + 20;
+
+	if (sizeof(rewrite_query_string_buffer) < strlen(message) + static_length) {
+		message[sizeof(rewrite_query_string_buffer) - static_length - 3] = '.';
+		message[sizeof(rewrite_query_string_buffer) - static_length - 2] = '.';
+		message[sizeof(rewrite_query_string_buffer) - static_length - 1] = '.';
+		message[sizeof(rewrite_query_string_buffer) - static_length - 0] = '\0';
+	}
+
+	rewrite_error_query_static(query_context, message, errcode);
+}
+
 static void run_clear_query(POOL_SESSION_CONTEXT* session_context, POOL_QUERY_CONTEXT* query_context)
 {
 	// TODO
@@ -413,7 +444,7 @@ static void run_and_rewrite_system_catalog_query(POOL_SESSION_CONTEXT* session_c
 	buffer = strcpy_capped(buffer, bufend - buffer, "'");
 
 	if (buffer == NULL) {
-		rewrite_error_query(query_context, "schema name too long", NULL);
+		rewrite_error_query_static(query_context, "schema name too long", NULL);
 		return;
 	}
 
@@ -425,7 +456,7 @@ static void run_and_rewrite_system_catalog_query(POOL_SESSION_CONTEXT* session_c
 	if (errmsg != NULL) {
 		rewrite_error_query(query_context, errmsg, errcode);
 	} else if (status != POOL_CONTINUE) {
-		rewrite_error_query(query_context, "Unknown execution error", NULL);
+		rewrite_error_query_static(query_context, "Unknown execution error", NULL);
 	}
 
 	/* build run_system_catalog_as_temp_table query */
@@ -447,7 +478,7 @@ static void run_and_rewrite_system_catalog_query(POOL_SESSION_CONTEXT* session_c
 	buffer = strcpy_capped(buffer, bufend - buffer, "')");
 
 	if (buffer == NULL) {
-		rewrite_error_query(query_context, "metadata too long", NULL);
+		rewrite_error_query_static(query_context, "metadata too long", NULL);
 		return;
 	}
 
@@ -459,7 +490,7 @@ static void run_and_rewrite_system_catalog_query(POOL_SESSION_CONTEXT* session_c
 	if (errmsg != NULL) {
 		rewrite_error_query(query_context, errmsg, errcode);
 	} else if (status != POOL_CONTINUE) {
-		rewrite_error_query(query_context, "Unknown execution error", NULL);
+		rewrite_error_query_static(query_context, "Unknown execution error", NULL);
 	}
 
 	/* rewrite query */
@@ -470,13 +501,47 @@ static void run_and_rewrite_system_catalog_query(POOL_SESSION_CONTEXT* session_c
 	buffer = strcpy_capped(buffer, bufend - buffer, PRESTO_RESULT_TABLE_NAME);
 
 	if (buffer == NULL) {
-		rewrite_error_query(query_context, "query too long", NULL);
+		rewrite_error_query_static(query_context, "query too long", NULL);
 		return;
 	}
 
 	do_replace_query(query_context, rewrite_query_string_buffer);
 }
 
+/*
+ * /\A\s*select\s*\*\s*from\s+(("[^\\"]*([\\"][^\\"]*)*")|[a-zA-Z_][a-zA-Z0-9_]*)(\.(("[^\\"]*([\\"][^\\"]*)*")|[a-zA-Z_][a-zA-Z0-9_]*))?\s*(;|\z)/.to_s
+ */
+#define AUTO_LIMIT_QUERY_PATTERN "\\A\\s*select\\s*\\*\\s*from\\s+((\"[^\\\\\"]*([\\\\\"][^\\\\\"]*)*\")|[a-zA-Z_][a-zA-Z0-9_]*)(\\.((\"[^\\\\\"]*([\\\\\"][^\\\\\"]*)*\")|[a-zA-Z_][a-zA-Z0-9_]*))?\\s*(;|\\z)"
+
+static bool match_auto_limit_pattern(const char* query)
+{
+	const char* errptr;
+	int erroffset;
+	pcre* pattern;
+	int ret;
+	int ovec[10];
+
+	pattern = pcre_compile(AUTO_LIMIT_QUERY_PATTERN,
+			PCRE_CASELESS | PCRE_NO_AUTO_CAPTURE | PCRE_UTF8, &errptr, &erroffset, NULL);
+	if (pattern == NULL) {
+		// TODO pcre pattern should be precompiled. see also pcre_study.
+		pool_error("match_auto_limit_pattern: invalid regexp %s at %d", errptr, erroffset);
+		pcre_free(pattern);
+		return false;
+	}
+
+	ret = pcre_exec(pattern, NULL, query, strlen(query), 0, 0, &ovec, sizeof(ovec));
+	if (ret < 0) {
+		// error. pattern didn't match in most of cases
+		pcre_free(pattern);
+		return false;
+	}
+
+	pcre_free(pattern);
+
+	return true;
+}
+
 static void run_and_rewrite_presto_query(POOL_SESSION_CONTEXT* session_context, POOL_QUERY_CONTEXT* query_context)
 {
 	char *buffer, *bufend;
@@ -519,11 +584,17 @@ static void run_and_rewrite_presto_query(POOL_SESSION_CONTEXT* session_context,
 		if (c != NULL) {
 			*c = save;
 		}
+
+		if (match_auto_limit_pattern(query)) {
+			// TODO send warning message to client
+			pool_debug("run_and_rewrite_presto_query: adding 'limit 1000' to a SELECT * query");
+			buffer = strcpy_capped(buffer, bufend - buffer, " limit 1000");
+		}
 	}
 	buffer = strcpy_capped(buffer, bufend - buffer, "')");
 
 	if (buffer == NULL) {
-		rewrite_error_query(query_context, "query too long", NULL);
+		rewrite_error_query_static(query_context, "query too long", NULL);
 		return;
 	}
 
@@ -536,7 +607,7 @@ static void run_and_rewrite_presto_query(POOL_SESSION_CONTEXT* session_context,
 		rewrite_error_query(query_context, errmsg, errcode);
 		return;
 	} else if (status != POOL_CONTINUE) {
-		rewrite_error_query(query_context, "Unknown execution error", NULL);
+		rewrite_error_query_static(query_context, "Unknown execution error", NULL);
 		return;
 	}
 
@@ -548,7 +619,7 @@ static void run_and_rewrite_presto_query(POOL_SESSION_CONTEXT* session_context,
 	buffer = strcpy_capped(buffer, bufend - buffer, PRESTO_RESULT_TABLE_NAME);
 
 	if (buffer == NULL) {
-		rewrite_error_query(query_context, "query too long", NULL);
+		rewrite_error_query_static(query_context, "query too long", NULL);
 		return;
 	}
 
@@ -564,7 +635,7 @@ void pool_where_to_send(POOL_QUERY_CONTEXT *query_context, char *query, Node *no
 	POOL_CONNECTION_POOL *backend;
 	int i;
 
-	const char* rewrite_error_message;
+	const char* static_error_message;
 	enum {
 		REWRITE_CLEAR,
 		REWRITE_SYSTEM_CATALOG,
@@ -676,7 +747,7 @@ void pool_where_to_send(POOL_QUERY_CONTEXT *query_context, char *query, Node *no
 				 */
 				if (TSTATE(backend, PRIMARY_NODE_ID) == 'I' ||
 					(!pool_is_writing_transaction() &&
-					 !pool_is_failed_transaction() &&
+					 !(MAJOR(backend) == PROTO_MAJOR_V3 && pool_is_failed_transaction()) &&
 					 pool_get_transaction_isolation() != POOL_SERIALIZABLE))
 				{
 					BackendInfo *bkinfo = pool_get_node_info(session_context->load_balance_node_id);
@@ -695,7 +766,7 @@ void pool_where_to_send(POOL_QUERY_CONTEXT *query_context, char *query, Node *no
 						pool_debug("prestogres: send_to_where: replication delay");
 						pool_set_node_to_be_sent(query_context, PRIMARY_NODE_ID);
 						rewrite_mode = REWRITE_ERROR;
-						rewrite_error_message = "unexpected replication delay";
+						static_error_message = "unexpected replication delay";
 					}
 
 					/*
@@ -776,16 +847,27 @@ void pool_where_to_send(POOL_QUERY_CONTEXT *query_context, char *query, Node *no
 					pool_debug("prestogres: send_to_where: invalid session state");
 					pool_set_node_to_be_sent(query_context, PRIMARY_NODE_ID);
 					rewrite_mode = REWRITE_ERROR;
-					rewrite_error_message = "invalid session state";
+					static_error_message = "invalid session state";
 				}
 			}
+			else if (query_context->is_parse_error)
+			{
+				/*
+				 * If failed to parse the query, run it on Presto because
+				 * it may include Presto's SQL syntax extensions.
+				 */
+				pool_debug("prestogres: send_to_where: parse-error");
+				pool_set_node_to_be_sent(query_context,
+						session_context->load_balance_node_id);
+				rewrite_mode = REWRITE_PRESTO;
+			}
 			else
 			{
 				/* Send to the primary only */
 				pool_debug("prestogres: send_to_where: non-select");
 				pool_set_node_to_be_sent(query_context, PRIMARY_NODE_ID);
 				rewrite_mode = REWRITE_ERROR;
-				rewrite_error_message = "only SELECT is supported";
+				static_error_message = "only SELECT is supported";
 			}
 		}
 	}
@@ -891,7 +973,7 @@ void pool_where_to_send(POOL_QUERY_CONTEXT *query_context, char *query, Node *no
 		break;
 
 	case REWRITE_ERROR:
-		rewrite_error_query(query_context, rewrite_error_message, NULL);
+		rewrite_error_query_static(query_context, static_error_message, NULL);
 		break;
 	}
 }

data/pgsql/prestogres.py

@@ -48,25 +48,22 @@ def _build_create_temp_table_sql(table_name, column_names, column_types):
     return create_sql
 
 # build CREATE TABLE statement
-def _build_create_table_sql(schema_name, table_name, column_names, column_types, not_nulls):
-    create_sql = "create table %s.%s (\n  " % (plpy.quote_ident(schema_name), plpy.quote_ident(table_name))
+def _build_alter_table_holder_sql(schema_name, table_name, column_names, column_types, not_nulls):
+    alter_sql = "alter table %s.%s \n  " % (plpy.quote_ident(schema_name), plpy.quote_ident(table_name))
 
     first = True
     for column_name, column_type, not_null in zip(column_names, column_types, not_nulls):
         if first:
             first = False
         else:
-            create_sql += ",\n  "
+            alter_sql += ",\n  "
 
-        create_sql += plpy.quote_ident(column_name)
-        create_sql += " "
-        create_sql += column_type
+        alter_sql += "add %s %s" % (plpy.quote_ident(column_name), column_type)
 
         if not_null:
-            create_sql += " not null"
+            alter_sql += " not null"
 
-    create_sql += "\n)"
-    return create_sql
+    return alter_sql
 
 # build INSERT INTO statement and string format to build VALUES (..), ...
 def _build_insert_into_sql(table_name, column_names):
@@ -231,6 +228,8 @@ def run_system_catalog_as_temp_table(server, user, catalog, schema, result_table
             statements = []
             schema_names = []
 
+            table_holder_id = 0
+
             for schema_name, tables in schemas.items():
                 if schema_name == "sys" or schema_name == "information_schema":
                     # skip system schemas
@@ -248,8 +247,17 @@ def run_system_catalog_as_temp_table(server, user, catalog, schema, result_table
                         column_types.append(_pg_table_type(column.type))
                         not_nulls.append(not column.nullable)
 
-                    create_sql = _build_create_table_sql(schema_name, table_name, column_names, column_types, not_nulls)
-                    statements.append(create_sql)
+                    # rename table holder into the schema
+                    statements.append("alter table prestogres_catalog.table_holder_%d set schema %s" % \
+                            (table_holder_id, plpy.quote_ident(schema_name)))
+                    statements.append("alter table %s.table_holder_%d rename to %s" % \
+                            (plpy.quote_ident(schema_name), table_holder_id, plpy.quote_ident(table_name)))
+
+                    # change columns
+                    alter_sql = _build_alter_table_holder_sql(schema_name, table_name, column_names, column_types, not_nulls)
+                    statements.append(alter_sql)
+
+                    table_holder_id += 1
 
             # cache expires after 60 seconds
             SchemaCacheEntry.set_cache(server, user, catalog, schema, schema_names, statements, time.time() + 60)
@@ -267,45 +275,50 @@ def run_system_catalog_as_temp_table(server, user, catalog, schema, result_table
             subxact = plpy.subtransaction()
             subxact.enter()
             try:
-                # delete all schemas excepting prestogres_catalog
+                # drop all schemas excepting prestogres_catalog, pg_catalog, information_schema, public
+                # and schema holders
                 sql = "select n.nspname as schema_name from pg_catalog.pg_namespace n" \
                       " where n.nspname not in ('prestogres_catalog', 'pg_catalog', 'information_schema', 'public')" \
+                      " and n.nspname not like 'prestogres_catalog_schema_holder_%'" \
                       " and n.nspname !~ '^pg_toast'"
                 for row in plpy.cursor(sql):
                     plpy.execute("drop schema %s cascade" % plpy.quote_ident(row["schema_name"]))
 
-                # delete all tables in prestogres_catalog
-                # relkind = 'r' takes only tables and skip views, indexes, etc.
-                sql = "select n.nspname as schema_name, c.relname as table_name from pg_catalog.pg_class c" \
-                      " left join pg_catalog.pg_namespace n on n.oid = c.relnamespace" \
-                      " where c.relkind in ('r')" \
-                      " and n.nspname in ('prestogres_catalog')" \
-                      " and n.nspname !~ '^pg_toast'"
-                for row in plpy.cursor(sql):
-                    plpy.execute("drop table %s.%s" % (plpy.quote_ident(row["schema_name"]), plpy.quote_ident(row["table_name"])))
-
-                # create schemas
+                # alter schema holders
+                schema_holder_id = 0
                 for schema_name in schema_names:
                     try:
-                        plpy.execute("create schema %s" % plpy.quote_ident(schema_name))
+                        plpy.execute("alter schema prestogres_catalog_schema_holder_%s rename to %s" % \
+                                (schema_holder_id, plpy.quote_ident(schema_name)))
+                        schema_holder_id += 1
                     except:
-                        # ignore error
+                        # ignore error?
                         pass
 
-                # create tables
+                # alter table holders in prestogres_catalog schema
                 for statement in statements:
                     plpy.execute(statement)
 
-                # run the actual query
+                # drop prestogres_catalog schema
+                plpy.execute("drop schema prestogres_catalog cascade")
+
+                # drop schema holders
+                sql = "select n.nspname as schema_name from pg_catalog.pg_namespace n" \
+                      " where n.nspname like 'prestogres_catalog_schema_holder_%'"
+                for row in plpy.cursor(sql):
+                    plpy.execute("drop schema %s" % plpy.quote_ident(row["schema_name"]))
+
+                # run the actual query and save result
                 metadata = plpy.execute(query)
                 column_names = metadata.colnames()
                 column_type_oids = metadata.coltypes()
                 result = map(lambda row: map(row.get, column_names), metadata)
 
-                # table schema
+                # save result schema
                 oid_to_type_name = _load_oid_to_type_name_mapping(column_type_oids)
                 column_types = map(oid_to_type_name.get, column_type_oids)
 
+                # store query cache
                 query_cache[query] = QueryResult(column_names, column_types, result)
 
             finally:
@@ -325,3 +338,12 @@ def run_system_catalog_as_temp_table(server, user, catalog, schema, result_table
         e.__class__.__module__ = "__main__"
         raise
 
+def create_schema_holders(count):
+    for i in range(count):
+        plpy.execute("drop schema if exists prestogres_catalog_schema_holder_%d cascade" % i)
+        plpy.execute("create schema prestogres_catalog_schema_holder_%d" % i)
+
+def create_table_holders(count):
+    for i in range(count):
+        plpy.execute("create table prestogres_catalog.table_holder_%d ()" % i)
+

data/pgsql/setup.sql

@@ -1,6 +1,8 @@
 
 create language "plpythonu";
 
+drop schema if exists prestogres_catalog cascade;
+
 create schema prestogres_catalog;
 
 create or replace function prestogres_catalog.run_presto_as_temp_table(
@@ -27,3 +29,19 @@ import prestogres
 prestogres.run_system_catalog_as_temp_table(server, user, catalog, schema, table_name, query)
 $$ language plpythonu;
 
+create or replace function prestogres_catalog.create_schema_holders("count" int)
+returns void as $$
+import prestogres
+prestogres.create_schema_holders(count);
+$$ language plpythonu;
+
+create or replace function prestogres_catalog.create_table_holders("count" int)
+returns void as $$
+import prestogres
+prestogres.create_table_holders(count);
+$$ language plpythonu;
+
+select prestogres_catalog.create_schema_holders(512);
+
+select prestogres_catalog.create_table_holders(2048);
+

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: prestogres
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.4.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-02-07 00:00:00.000000000 Z
+date: 2014-03-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -479,18 +479,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: -1048444669451031740
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: -1048444669451031740
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.23
@@ -498,3 +492,4 @@ signing_key:
 specification_version: 3
 summary: Presto PostgreSQL protocol gateway
 test_files: []
+has_rdoc: false