prestogres 0.2.0 → 0.3.0

data/.gitignore

@@ -1,4 +1,7 @@
 *~
 *.pyc
 .bundle
+.so
 pkg
+vendor/bundle
+Gemfile.lock

data/ChangeLog

@@ -1,4 +1,18 @@
 
+2014-02-07 version 0.3.0:
+
+* Fixed system catalog queries to support multiple schemas
+* Removed presto_schema parameter from pgpool.conf
+* Use connecting database name as the default schema name
+* Updated type mapping to map varchar type of Presto to varchar(100)
+* Implemented query cache for system catalog queries
+* Improved prestogres-setup command to not listen on TCP socket
+* Improved prestogres-setup command to support -P option
+* Fixed build scripts to work with bundler
+* Fixed build scripts to work with RubyGems >= 1.8.26
+* Fixed "rake install" task
+
+
 2014-01-24 version 0.2.0:
 
 * Authentication mechanisms support pg_database and pg_user options to
@@ -10,6 +24,7 @@
 * Queries to system catalogs delete created tables by rollbacking
 subtransaction so that other sessions does not conflict
 
+
 2014-01-15 version 0.1.0:
 
 * First release

data/README.md

@@ -1,11 +1,11 @@
 # ![Prestogres](https://gist.github.com/frsyuki/8328440/raw/6c3a19b7132fbbf975155669f308854f70fff1e8/prestogres.png)
 ## PostgreSQL protocol gateway for Presto
 
-**Prestogres** is a gateway server that allows clients to use PostgreSQL protocol to run Presto queries.
+**Prestogres** is a gateway server that allows clients to use PostgreSQL protocol to run queries on Presto.
 
 * [Presto, a distributed SQL query engine for big data](https://github.com/facebook/presto)
 
-With Prestogres, you can use PostgreSQL clients to run queries on Presto:
+You can use any PostgreSQL clients (see also *Limitation* section):
 
 * `psql` command
 * [PostgreSQL ODBC driver](http://psqlodbc.projects.pgfoundry.org/)
@@ -16,6 +16,9 @@ Prestogres also offers password-based authentication and SSL.
 
 ## How it works?
 
+Prestogres uses modified **[pgpool-II](http://www.pgpool.net/)** to rewrite queries before sending them to PostgreSQL.
+pgpool-II is originally an open-source middleware to provide connection pool and other features to PostgreSQL.
+
 ```
        PostgreSQL protocol                     Presto protocol (HTTP)
             /                                      /
@@ -37,6 +40,7 @@ Prestogres also offers password-based authentication and SSL.
 
 * Extended query is not supported ([PostgreSQL Frontend/Backend Protocol](http://www.postgresql.org/docs/9.3/static/protocol.html))
   * ODBC driver needs to set **UseServerSidePrepare=0** (Server side prepare: no) property
+  * ODBC driver needs to use "Unicode" mode
   * JDBC driver needs to set **protocolVersion=2** property
 * DECLARE/FETCH is not supported
 
@@ -46,41 +50,38 @@ Prestogres also offers password-based authentication and SSL.
 $ gem install prestogres --no-ri --no-rdoc
 ```
 
-Prestogres package installs patched pgpool-II but doesn't install PostgreSQL. You need to install PostgreSQL server (with python support) separately.
+Prestogres package installs patched pgpool-II but doesn't install PostgreSQL. You need to install PostgreSQL >= 9.3 (with python support) separately.
 
 * If you don't have **gem** command, install Ruby >= 1.9.0 first
 * If installation failed, you may need to install following packages using apt or yum:
   * basic toolchain (gcc, make, etc.)
-  * OpenSSL (Debian/Ubuntu: **libssl-dev**, RedHat/CentOS: **openssl-dev**)
-  * PostgreSQL server (Debian/Ubuntu: **postgresql-server-dev**, RedHat/CentOS: **postgresql-devel**)
+  * OpenSSL dev package (Debian/Ubuntu: **libssl-dev**, RedHat/CentOS: **openssl-dev**)
+  * PostgreSQL server dev package (Debian/Ubuntu: **postgresql-server-dev-9.3**, RedHat/CentOS: **postgresql-devel**)
+
+### Installation FAQ
+
 
 ## Runing servers
 
-You need to run 2 server programs: pgpool-II and PostgreSQL. You can use `prestogres` command to setup & run them.
+You need to run 2 server programs: pgpool-II and PostgreSQL.
+You can use `prestogres` command to setup & run them as following:
 
-1. Create a data directory:
-```
+```sh
+# 1. Create a data directory:
 $ prestogres -D pgdata setup
-```
 
-2. Configure **presto_server** and **presto_catalog** parameters at least in pgpool.conf file:
-```
+# 2. Configure presto_server and presto_catalog parameters at least in pgpool.conf file:
 $ vi ./pgdata/pgpool/pgpool.conf
-```
 
-3. Run patched pgpool-II:
-```
+# 3. Run patched pgpool-II:
 $ prestogres -D pgdata pgpool
-```
 
-4. Run PostgreSQL:
-```
+# 4. Run PostgreSQL:
 $ prestogres -D pgdata pg_ctl start
-```
 
-5. Finally, you can connect to pgpool-II using `psql` command:
-```
+# 5. Finally, you can connect to pgpool-II using `psql` command:
 $ psql -h localhost -p 9900 -U pg postgres
+> SELECT * FROM sys.node;
 ```
 
 If configuration is correct, you can run `SELECT * FROM sys.node;` query. Otherwise, see log files in **./pgdata/log/** directory.
@@ -111,7 +112,6 @@ Following parameters are unique to Prestogres:
 
 * **presto_server**: Default address:port of Presto server.
 * **presto_catalog**: Default catalog (connector) name of Presto such as `hive-cdh4`, `hive-hadoop1`, etc.
-* **presto_schema**: Default schema name of Presto. You can read other schemas by qualified name like `FROM myschema.table1`
 * **presto_external_auth_prog**: Default path to an external authentication program used by `prestogres_external` authentication moethd. See following Authentication section for details.
 
 You can overwrite these parameters for each connecting users. See also following *pool_hba.conf* section.
@@ -124,10 +124,10 @@ See [sample pool_hba.conf file](https://github.com/treasure-data/prestogres/blob
 
 ```conf
 # TYPE   DATABASE   USER   CIDR-ADDRESS                  METHOD                OPTIONS
-host     all        all    127.0.0.1/32                  trust
-host     all        all    127.0.0.1/32,192.168.0.0/16   prestogres_md5
-host     altdb      pg     0.0.0.0/0                     prestogres_md5        server:localhost:8190,user:prestogres
-host     all        all    0.0.0.0/0                     prestogres_external   auth_prog:/opt/prestogres/auth.py
+host     postgres   pg     127.0.0.1/32                  trust
+host     postgres   pg     127.0.0.1/32,192.168.0.0/16   prestogres_md5
+host     altdb      pg     0.0.0.0/0                     prestogres_md5        server:localhost:8190,pg_database:postgres
+host     all        all    0.0.0.0/0                     prestogres_external   auth_prog:/opt/prestogres/auth.py,pg_database:postgres,pg_user:pg
 ```
 
 See also *Creating database* section.
@@ -145,8 +145,8 @@ In pool_hba.conf file, you can set following options to OPTIONS field:
 
 * **server**: Address:port of Presto server, which overwrites `presto_servers` parameter in pgpool.conf.
 * **catalog**: Catalog (connector) name of Presto, which overwrites `presto_catalog` parameter in pgpool.conf.
-* **schema**: Schema name of Presto, which overwrites `presto_schema` parameter in pgpool.conf.
-* **user**: User name to run queries on Presto. By default, Prestogres uses the same user name used to login to pgpool-II.
+* **schema**: Default schema name of Presto. By default, Prestogres uses the same name with the database name used to login to pgpool-II. Following `pg_database` parameter doesn't overwrite affect this parameter.
+* **user**: User name to run queries on Presto. By default, Prestogres uses the same user name used to login to pgpool-II. Following `pg_user` parameter doesn't overwrite affect this parameter.
 * **pg_database**: Overwrite database to connect to PostgreSQL.
 * **pg_user**: Overwrite user name to connect to PostgreSQL.
 
@@ -223,7 +223,7 @@ commands:
 
 ## Development
 
-To install git HEAD, use following command to build:
+To install git HEAD, use following commands to build:
 
 ```sh
 # 1. clone prestogres repository:
@@ -239,7 +239,6 @@ $ bundle
 $ bundle exec rake
 
 # 4. install the created package:
-$ gem install --no-ri --no-rdoc pkg/prestogres-0.1.0.gem
+$ gem install --no-ri --no-rdoc pkg/prestogres-*.gem
 # if this command failed, you may need to install toolchain (gcc, etc.) to build pgpool-II
 ```
-

data/Rakefile

@@ -4,9 +4,10 @@ Bundler::GemHelper.install_tasks
 require 'rake/extensiontask'
 
 spec = eval File.read("prestogres.gemspec")
-Rake::ExtensionTask.new('prestogres', spec) do |ext|
+Rake::ExtensionTask.new('prestogres_config', spec) do |ext|
   ext.cross_compile = true
-  ext.lib_dir = File.join(*['lib', 'prestogres', ENV['FAT_DIR']].compact)
+  ext.lib_dir = 'lib'
+  ext.ext_dir = 'ext'
   #ext.cross_platform = 'i386-mswin32'
 end
 

data/VERSION

@@ -1 +1 @@
-0.2.0
+0.3.0

data/bin/prestogres

@@ -7,7 +7,7 @@ config = {
 }
 
 def usage(error=nil)
-  puts "usage: #{File.basename($0)} -D <data dir> <command>"
+  puts "usage: #{File.basename($0)} -D <data dir> [-P <postgresql bin path>] <command>"
   puts "commands:"
   puts "  setup                 setup <data dir>"
   puts "  pgpool                start pgpool as a daemon process"
@@ -38,6 +38,8 @@ ARGV.each_with_index do |a,i|
   case a
   when "-D"
     config_key = :data_dir
+  when "-P"
+    config_key = :bin_path
   when "-h", "-?", "--help"
     config[:command] = :help
   when "setup"
@@ -73,6 +75,8 @@ setup_params = {
  unix_socket_directory: nil,
 }
 
+ENV['PATH'] = "#{ENV['PATH']}:#{config[:bin_path]}"
+
 case config[:command]
 when :help
   usage nil
@@ -87,15 +91,11 @@ when :setup
 
   require "erb"
   require "fileutils"
+  require "shellwords"
 
-  def run_setup_command(cmdline)
-    if cmdline.is_a?(Array)
-      puts "setup> #{cmdline.join(' ')}"
-      system *cmdline
-    else
-      puts "setup> #{cmdline}"
-      system cmdline
-    end
+  def setup_cmd(cmdline)
+    puts "setup> #{cmdline}"
+    system cmdline
     unless $?.success?
       puts "******************"
       puts "** setup failed **"
@@ -114,44 +114,84 @@ when :setup
     end
   end
 
+  def se(raw)
+    Shellwords.escape(raw)
+  end
+
+  `which initdb && which postgres`
+  unless $?.success?
+    puts ""
+    puts "Can't find initdb or postgres command. Check following items:"
+    puts ""
+    puts "  * You need to install PostgreSQL server. Use apt or yum to install postgresql package."
+    puts "  * You need to set -P option to specify path to the commands. Example:"
+    puts ""
+    puts "    $ prestogres -D data_dir -P /usr/lib/postgresql/9.1/bin setup"
+    puts ""
+    exit 1
+  end
+
+  # src files
   config_src_dir = File.join(File.dirname(__FILE__), "..", "config")
+  postgresql_conf_path = File.join(config_src_dir, "postgresql.conf")
   pgsql_dir = File.join(File.dirname(__FILE__), "..", "pgsql")
+  setup_sql_path = File.join(pgsql_dir, "setup.sql")
 
   puts "Setting up '#{config[:data_dir]}'..."
   data_dir = File.expand_path(config[:data_dir])
 
-  @config = setup_params  # used by erb files
-  @config[:pgpool_pid_file] = File.join(data_dir, "run", "pgpool.pid")
-  @config[:pgpool_status_dir] = File.join(data_dir, "run")
-  @config[:unix_socket_directory] = File.join(data_dir, "run")
-
   # log & socket dirs
-  FileUtils.mkdir_p File.join(data_dir, "log")
-  FileUtils.mkdir_p File.join(data_dir, "run")
+  log_dir = File.join(data_dir, "log")
+  run_dir = File.join(data_dir, "run")
+  FileUtils.mkdir_p log_dir
+  FileUtils.mkdir_p run_dir
+
+  pgpool_dir = File.join(data_dir, "pgpool")
+  postgres_dir = File.join(data_dir, "postgres")
+
+  @config = setup_params  # used by erb files
+  @config[:pgpool_pid_file] = File.join(run_dir, "pgpool.pid")
+  @config[:pgpool_status_dir] = run_dir
+  @config[:unix_socket_directory] = run_dir
 
   # pgpool
-  FileUtils.mkdir_p File.join(data_dir, "pgpool")
+  FileUtils.mkdir_p pgpool_dir
   %w[pcp.conf.sample pgpool.conf pool_hba.conf pool_passwd].each do |fname|
-    File.open(File.join(data_dir, "pgpool", fname), "w") {|f|
-      f.write ERB.new(File.read(File.join(config_src_dir, fname))).result
+    # run ERB for each config files
+    dst_path = File.join(pgpool_dir, fname)
+    src_path = File.join(config_src_dir, fname)
+    File.open(dst_path, "w") {|f|
+      f.write ERB.new(File.read(src_path)).result
     }
   end
 
   # postgresql
-  run_setup_command %w[initdb -U pg --no-locale -E UNICODE] + [File.join(data_dir, "postgres")]
+  # 1. initdb postgres_dir
+  setup_cmd "initdb -U pg --no-locale -E UNICODE #{se(postgres_dir)}"
   File.open(File.join(data_dir, "postgres", "postgresql.conf"), "a") {|f|
-    f.write ERB.new(File.read(File.join(config_src_dir, "postgresql.conf"))).result
+    f.write ERB.new(File.read(postgresql_conf_path)).result
   }
 
-  # setup functions
-  run_setup_command %w[pg_ctl -D] + [File.join(data_dir, 'postgres')] + %w[start]
+  # 2. postgres -D postgres_dir -c listen_addresses=
+  # start postgres without listening on TCP
+  postgres_cmd = "postgres -D #{se(postgres_dir)} -c listen_addresses="
+  puts "setup> #{postgres_cmd}"
+  pid = spawn postgres_cmd
+
+  # 3. psql < pgsql/setup.sql
   begin
-    sleep 1
+    # waits for spinup of postgres
+    sleep 2
+    puts "initializing database..."
+    sleep 8
 
-    run_setup_command "psql -h 127.0.0.1 -p #{setup_params[:backend_port]} -U pg postgres < #{File.join(pgsql_dir, "setup.sql")}"
+    psql_cmd = "psql -h #{se(@config[:unix_socket_directory])} -p #{@config[:backend_port]} -U pg postgres"
+    setup_cmd "#{psql_cmd} < #{se(setup_sql_path)}"
   ensure
-    run_setup_command %w[pg_ctl -D] + [File.join(data_dir, 'postgres')] + %w[stop]
+    # 4. shutdown postgres
+    Process.kill(:SIGQUIT, pid)
   end
+  Process.waitpid(pid)
 
   puts <<EOF
 
@@ -203,7 +243,9 @@ when :passwd
   binary = File.join(prefix, "bin", 'pg_md5')
 
   data_dir = config[:data_dir]
-  args = ["-f", File.join(data_dir, "pgpool", "pgpool.conf"), "-p", "-m", "-u"].concat(args)
+  pgpool_conf_path = File.join(data_dir, "pgpool", "pgpool.conf")
+
+  args = ["-f", pgpool_conf_path, "-p", "-m", "-u"].concat(args)
 
   puts "#{binary} #{args.join(' ')}"
   system binary, *args

data/config/postgresql.conf

@@ -1,2 +1,2 @@
 port = <%= @config[:backend_port] %>
-unix_socket_directory = '<%= @config[:unix_socket_directory] %>'
+unix_socket_directories = '<%= @config[:unix_socket_directory] %>'

data/ext/depend

@@ -1,8 +1,16 @@
 # vim: ft=make noexpandtab
 
-INSTALL_DIR := $(if $(rubyarchdir),$(rubyarchdir),$(RUBYARCHDIR))/prestogres
+# normalize rubyarchdir which is not available depending on rake or gem
+narchdir = $(if $(rubyarchdir),$(rubyarchdir),$(RUBYARCHDIR))
 
-CFLAGS += -DPRESTOGRES_PREFIX="\"$(INSTALL_DIR)\""
+# hack for RubyGems >= 1.8.26
+final_archdir = $(if $(findstring .gem.,$(narchdir)),$(dir $(narchdir)),$(narchdir))
+
+INSTALL_DATA = $(realpath $(final_archdir))/prestogres
+
+PGPOOL2_PATH = $(realpath $(srcdir)/../pgpool2)
+
+CFLAGS += -DPRESTOGRES_PREFIX="\"$(INSTALL_DATA)\""
 
 all: pgpool2
 
@@ -12,10 +20,10 @@ install: install-pgpool2
 
 pgpool2:
 	mkdir -p build
-	cd build && ../../pgpool2/configure \
-		CFLAGS="-I$(CURDIR)/../pgpool2" \
+	cd build && "$(PGPOOL2_PATH)/configure" \
+		CFLAGS="-I$(PGPOOL2_PATH)" \
 		--with-openssl \
-		--prefix="$(INSTALL_DIR)"
+		--prefix="$(INSTALL_DATA)"
 	cd build && make sysconfdir=/opt/prestogres/etc
 
 clean-pgpool2:

data/pgpool2/child.c

@@ -295,6 +295,9 @@ void do_child(int unix_fd, int inet_fd)
 			goto retry_startup;
 		}
 
+		/* this should run before ClientAuthentication */
+		pool_prestogres_set_defaults(sp);
+
 		if (pool_config->enable_pool_hba)
 		{
 			/*
@@ -332,9 +335,6 @@ void do_child(int unix_fd, int inet_fd)
 			}
 		}
 
-		/* this should run after ClientAuthentication */
-		pool_prestogres_init_session(frontend);
-
 		/*
 		 * Ok, negotiation with frontend has been done. Let's go to the
 		 * next step.  Connect to backend if there's no existing

data/pgpool2/pool.h

@@ -652,6 +652,6 @@ extern const char* presto_external_auth_prog;
 int send_md5auth_request(POOL_CONNECTION *frontend, int protoMajor, char *salt);
 int read_password_packet(POOL_CONNECTION *frontend, int protoMajor, 	char *password, int *pwdSize);
 
-void pool_prestogres_init_session(POOL_CONNECTION *frontend);
+void pool_prestogres_set_defaults(StartupPacket *sp);
 
 #endif /* POOL_H */

data/pgpool2/pool_config.c

@@ -1896,7 +1896,6 @@ int pool_init_config(void)
 
 	pool_config->presto_server = "";
 	pool_config->presto_catalog = "";
-	pool_config->presto_schema = "default";
 	pool_config->presto_external_auth_prog = NULL;
 
 	pool_config->replication_mode = 0;
@@ -2599,24 +2598,6 @@ int pool_get_config(char *confpath, POOL_CONFIG_CONTEXT context)
 			}
 			pool_config->presto_catalog = str;
 		}
-		else if (!strcmp(key, "presto_schema") && CHECK_CONTEXT(INIT_CONFIG, context))
-		{
-			char *str;
-
-			if (token != POOL_STRING && token != POOL_UNQUOTED_STRING && token != POOL_KEY)
-			{
-				PARSE_ERROR();
-				fclose(fd);
-				return(-1);
-			}
-			str = extract_string(yytext, token);
-			if (str == NULL)
-			{
-				fclose(fd);
-				return(-1);
-			}
-			pool_config->presto_schema = str;
-		}
 		else if (!strcmp(key, "presto_external_auth_prog") && CHECK_CONTEXT(INIT_CONFIG, context))
 		{
 			char *str;

data/pgpool2/pool_hba.c

@@ -1709,23 +1709,23 @@ static POOL_STATUS pool_prestogres_hba_auth_external(POOL_CONNECTION *frontend)
 	return POOL_CONTINUE;
 }
 
-void pool_prestogres_init_session(POOL_CONNECTION *frontend)
+void pool_prestogres_set_defaults(StartupPacket *sp)
 {
     if (presto_server == NULL) {
         presto_server = pool_config->presto_server;
     }
     if (presto_user == NULL) {
-        presto_user = frontend->username;
+        presto_user = strdup(sp->user);
     }
     if (presto_catalog == NULL) {
         presto_catalog = pool_config->presto_catalog;
     }
     if (presto_schema == NULL) {
-        presto_schema = pool_config->presto_schema;
+        presto_schema = strdup(sp->database);
     }
-    pool_debug("pool_prestogres_init_session: presto_server: %s", presto_server);
-    pool_debug("pool_prestogres_init_session: presto_user: %s", presto_user);
-    pool_debug("pool_prestogres_init_session: presto_catalog: %s", presto_catalog);
-    pool_debug("pool_prestogres_init_session: presto_schema: %s", presto_schema);
+    pool_debug("pool_prestogres_set_defaults: presto_server: %s", presto_server);
+    pool_debug("pool_prestogres_set_defaults: presto_user: %s", presto_user);
+    pool_debug("pool_prestogres_set_defaults: presto_catalog: %s", presto_catalog);
+    pool_debug("pool_prestogres_set_defaults: presto_schema: %s", presto_schema);
 }
 

data/pgpool2/pool_query_context.c

@@ -404,7 +404,31 @@ static void run_and_rewrite_system_catalog_query(POOL_SESSION_CONTEXT* session_c
 	POOL_CONNECTION_POOL *backend = session_context->backend;
 	con = CONNECTION(backend, session_context->load_balance_node_id);
 
-	/* build query */
+	/* build SET query */
+	buffer = rewrite_query_string_buffer;
+	bufend = buffer + sizeof(rewrite_query_string_buffer);
+
+	buffer = strcpy_capped(buffer, bufend - buffer, "set search_path to E'");
+	buffer = strcpy_capped_escaped(buffer, bufend - buffer, presto_schema, "'\\");
+	buffer = strcpy_capped(buffer, bufend - buffer, "'");
+
+	if (buffer == NULL) {
+		rewrite_error_query(query_context, "schema name too long", NULL);
+		return;
+	}
+
+	/* run SET query */
+	status = do_query_or_get_error_message(con,
+			rewrite_query_string_buffer, &res, MAJOR(backend), &errmsg, &errcode);
+	free_select_result(res);
+
+	if (errmsg != NULL) {
+		rewrite_error_query(query_context, errmsg, errcode);
+	} else if (status != POOL_CONTINUE) {
+		rewrite_error_query(query_context, "Unknown execution error", NULL);
+	}
+
+	/* build run_system_catalog_as_temp_table query */
 	buffer = rewrite_query_string_buffer;
 	bufend = buffer + sizeof(rewrite_query_string_buffer);
 
@@ -415,6 +439,8 @@ static void run_and_rewrite_system_catalog_query(POOL_SESSION_CONTEXT* session_c
 	buffer = strcpy_capped(buffer, bufend - buffer, "', E'");
 	buffer = strcpy_capped_escaped(buffer, bufend - buffer, presto_catalog, "'\\");
 	buffer = strcpy_capped(buffer, bufend - buffer, "', E'");
+	buffer = strcpy_capped_escaped(buffer, bufend - buffer, presto_schema, "'\\");
+	buffer = strcpy_capped(buffer, bufend - buffer, "', E'");
 	buffer = strcpy_capped_escaped(buffer, bufend - buffer, PRESTO_RESULT_TABLE_NAME, "'\\");
 	buffer = strcpy_capped(buffer, bufend - buffer, "', E'");
 	buffer = strcpy_capped_escaped(buffer, bufend - buffer, query_context->original_query, "'\\");
@@ -425,7 +451,7 @@ static void run_and_rewrite_system_catalog_query(POOL_SESSION_CONTEXT* session_c
 		return;
 	}
 
-	/* run query */
+	/* run run_system_catalog_as_temp_table query */
 	status = do_query_or_get_error_message(con,
 			rewrite_query_string_buffer, &res, MAJOR(backend), &errmsg, &errcode);
 	free_select_result(res);

data/pgsql/prestogres.py

@@ -6,7 +6,7 @@ import time
 # convert Presto query result type to PostgreSQL type
 def _pg_result_type(presto_type):
     if presto_type == "varchar":
-        return "text"
+        return "varchar(255)"
     elif presto_type == "bigint":
         return "bigint"
     elif presto_type == "boolean":
@@ -19,7 +19,7 @@ def _pg_result_type(presto_type):
 # convert Presto type to PostgreSQL type
 def _pg_table_type(presto_type):
     if presto_type == "varchar":
-        return "varchar(100)"
+        return "varchar(255)"
     elif presto_type == "bigint":
         return "bigint"
     elif presto_type == "boolean":
@@ -30,7 +30,7 @@ def _pg_table_type(presto_type):
         raise Exception("unknown table column type: " + plpy.quote_ident(presto_type))
 
 # build CREATE TEMPORARY TABLE statement
-def _build_create_temp_table_sql(table_name, column_names, column_types, not_nulls=None):
+def _build_create_temp_table_sql(table_name, column_names, column_types):
     create_sql = "create temporary table %s (\n  " % plpy.quote_ident(table_name)
 
     first = True
@@ -44,9 +44,26 @@ def _build_create_temp_table_sql(table_name, column_names, column_types, not_nul
         create_sql += " "
         create_sql += column_type
 
-        # TODO not null
-        #if not column.nullable:
-        #    create_sql += " not null"
+    create_sql += "\n)"
+    return create_sql
+
+# build CREATE TABLE statement
+def _build_create_table_sql(schema_name, table_name, column_names, column_types, not_nulls):
+    create_sql = "create table %s.%s (\n  " % (plpy.quote_ident(schema_name), plpy.quote_ident(table_name))
+
+    first = True
+    for column_name, column_type, not_null in zip(column_names, column_types, not_nulls):
+        if first:
+            first = False
+        else:
+            create_sql += ",\n  "
+
+        create_sql += plpy.quote_ident(column_name)
+        create_sql += " "
+        create_sql += column_type
+
+        if not_null:
+            create_sql += " not null"
 
     create_sql += "\n)"
     return create_sql
@@ -100,21 +117,28 @@ class SchemaCache(object):
         self.server = None
         self.user = None
         self.catalog = None
+        self.schema = None
         self.schema_names = None
         self.statements = None
         self.expire_time = None
+        self.query_cache = {}
 
-    def is_cached(self, server, user, catalog, current_time):
-        return self.server == server and self.user == user and self.catalog == catalog \
+    def is_cached(self, server, user, catalog, schema, current_time):
+        return self.server == server and self.user == user \
+               and self.catalog == catalog and self.schema == schema \
                and self.statements is not None and current_time < self.expire_time
 
-    def set_cache(self, server, user, catalog, schema_names, statements, expire_time):
+    def set_cache(self, server, user, catalog, schema, schema_names, statements, expire_time):
         self.server = server
         self.user = user
         self.catalog = catalog
+        self.schema = schema
         self.schema_names = schema_names
         self.statements = statements
         self.expire_time = expire_time
+        self.query_cache = {}
+
+QueryResult = namedtuple("QueryResult", ("column_names", "column_types", "result"))
 
 OidToTypeNameMapping = {}
 
@@ -171,14 +195,15 @@ def run_presto_as_temp_table(server, user, catalog, schema, result_table, query)
         e.__class__.__module__ = "__main__"
         raise
 
-def run_system_catalog_as_temp_table(server, user, catalog, result_table, query):
+def run_system_catalog_as_temp_table(server, user, catalog, schema, result_table, query):
     try:
-        client = presto_client.Client(server=server, user=user, catalog=catalog, schema="default")
+        client = presto_client.Client(server=server, user=user, catalog=catalog, schema=schema)
 
         # create SQL statements which put data to system catalogs
-        if SchemaCacheEntry.is_cached(server, user, catalog, time.time()):
+        if SchemaCacheEntry.is_cached(server, user, catalog, schema, time.time()):
             schema_names = SchemaCacheEntry.schema_names
             statements = SchemaCacheEntry.statements
+            query_cache = SchemaCacheEntry.query_cache
 
         else:
             # get table list
@@ -223,57 +248,69 @@ def run_system_catalog_as_temp_table(server, user, catalog, result_table, query)
                         column_types.append(_pg_table_type(column.type))
                         not_nulls.append(not column.nullable)
 
-                    create_sql = _build_create_temp_table_sql(table_name, column_names, column_types, not_nulls)
+                    create_sql = _build_create_table_sql(schema_name, table_name, column_names, column_types, not_nulls)
                     statements.append(create_sql)
 
-            # cache expires after 10 seconds
-            SchemaCacheEntry.set_cache(server, user, catalog, schema_names, statements, time.time() + 10)
+            # cache expires after 60 seconds
+            SchemaCacheEntry.set_cache(server, user, catalog, schema, schema_names, statements, time.time() + 60)
+            query_cache = {}
 
-        # enter subtransaction to rollback tables right after running the query
-        subxact = plpy.subtransaction()
-        subxact.enter()
-        try:
-            # delete all schemas excepting prestogres_catalog
-            sql = "select n.nspname as schema_name from pg_catalog.pg_namespace n" \
-                  " where n.nspname not in ('prestogres_catalog', 'pg_catalog', 'information_schema', 'public')" \
-                  " and n.nspname !~ '^pg_toast'"
-            for row in plpy.cursor(sql):
-                plpy.execute("drop schema %s cascade" % plpy.quote_ident(row["schema_name"]))
-
-            # delete all tables in prestogres_catalog
-            # relkind = 'r' takes only tables and skip views, indexes, etc.
-            sql = "select n.nspname as schema_name, c.relname as table_name from pg_catalog.pg_class c" \
-                  " left join pg_catalog.pg_namespace n on n.oid = c.relnamespace" \
-                  " where c.relkind in ('r')" \
-                  " and n.nspname in ('prestogres_catalog')" \
-                  " and n.nspname !~ '^pg_toast'"
-            for row in plpy.cursor(sql):
-                plpy.execute("drop table %s.%s" % (plpy.quote_ident(row["schema_name"]), plpy.quote_ident(row["table_name"])))
-
-            # create schemas
-            for schema_name in schema_names:
-                try:
-                    plpy.execute("create schema %s" % plpy.quote_ident(schema_name))
-                except:
-                    # ignore error
-                    pass
-
-            # create tables
-            for statement in statements:
-                plpy.execute(statement)
-
-            # run the actual query
-            metadata = plpy.execute(query)
-            result = map(lambda row: row.values(), metadata)
+        query_result = query_cache.get(query)
 
-        finally:
-            # rollback subtransaction
-            subxact.exit("rollback subtransaction", None, None)
+        if query_result:
+            column_names = query_result.column_names
+            column_types = query_result.column_types
+            result = query_result.result
 
-        # table schema
-        oid_to_type_name = _load_oid_to_type_name_mapping(metadata.coltypes())
-        column_names = metadata.colnames()
-        column_types = map(oid_to_type_name.get, metadata.coltypes())
+        else:
+            # enter subtransaction to rollback tables right after running the query
+            subxact = plpy.subtransaction()
+            subxact.enter()
+            try:
+                # delete all schemas excepting prestogres_catalog
+                sql = "select n.nspname as schema_name from pg_catalog.pg_namespace n" \
+                      " where n.nspname not in ('prestogres_catalog', 'pg_catalog', 'information_schema', 'public')" \
+                      " and n.nspname !~ '^pg_toast'"
+                for row in plpy.cursor(sql):
+                    plpy.execute("drop schema %s cascade" % plpy.quote_ident(row["schema_name"]))
+
+                # delete all tables in prestogres_catalog
+                # relkind = 'r' takes only tables and skip views, indexes, etc.
+                sql = "select n.nspname as schema_name, c.relname as table_name from pg_catalog.pg_class c" \
+                      " left join pg_catalog.pg_namespace n on n.oid = c.relnamespace" \
+                      " where c.relkind in ('r')" \
+                      " and n.nspname in ('prestogres_catalog')" \
+                      " and n.nspname !~ '^pg_toast'"
+                for row in plpy.cursor(sql):
+                    plpy.execute("drop table %s.%s" % (plpy.quote_ident(row["schema_name"]), plpy.quote_ident(row["table_name"])))
+
+                # create schemas
+                for schema_name in schema_names:
+                    try:
+                        plpy.execute("create schema %s" % plpy.quote_ident(schema_name))
+                    except:
+                        # ignore error
+                        pass
+
+                # create tables
+                for statement in statements:
+                    plpy.execute(statement)
+
+                # run the actual query
+                metadata = plpy.execute(query)
+                column_names = metadata.colnames()
+                column_type_oids = metadata.coltypes()
+                result = map(lambda row: map(row.get, column_names), metadata)
+
+                # table schema
+                oid_to_type_name = _load_oid_to_type_name_mapping(column_type_oids)
+                column_types = map(oid_to_type_name.get, column_type_oids)
+
+                query_cache[query] = QueryResult(column_names, column_types, result)
+
+            finally:
+                # rollback subtransaction
+                subxact.exit("rollback subtransaction", None, None)
 
         create_sql = _build_create_temp_table_sql(result_table, column_names, column_types)
         insert_sql, values_sql_format = _build_insert_into_sql(result_table, column_names)

data/pgsql/setup.sql

@@ -19,10 +19,11 @@ create or replace function prestogres_catalog.run_system_catalog_as_temp_table(
     "server" text,
     "user" text,
     "catalog" text,
+    "schema" text,
     "table_name" text,
     "query" text)
 returns void as $$
 import prestogres
-prestogres.run_system_catalog_as_temp_table(server, user, catalog, table_name, query)
+prestogres.run_system_catalog_as_temp_table(server, user, catalog, schema, table_name, query)
 $$ language plpythonu;
 

metadata

@@ -1,55 +1,62 @@
 --- !ruby/object:Gem::Specification
 name: prestogres
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
+  prerelease: 
 platform: ruby
 authors:
 - Sadayuki Furuhashi
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-01-25 00:00:00.000000000 Z
+date: 2014-02-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
+    none: false
     requirements:
-    - - "~>"
+    - - ~>
       - !ruby/object:Gem::Version
         version: '1.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
+    none: false
     requirements:
-    - - "~>"
+    - - ~>
       - !ruby/object:Gem::Version
         version: '1.0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
+    none: false
     requirements:
-    - - "~>"
+    - - ~>
       - !ruby/object:Gem::Version
         version: 0.9.2
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
+    none: false
     requirements:
-    - - "~>"
+    - - ~>
       - !ruby/object:Gem::Version
         version: 0.9.2
 - !ruby/object:Gem::Dependency
   name: rake-compiler
   requirement: !ruby/object:Gem::Requirement
+    none: false
     requirements:
-    - - "~>"
+    - - ~>
       - !ruby/object:Gem::Version
         version: 0.8.3
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
+    none: false
     requirements:
-    - - "~>"
+    - - ~>
       - !ruby/object:Gem::Version
         version: 0.8.3
 description: Presto PostgreSQL protocol gateway
@@ -61,10 +68,9 @@ extensions:
 - ext/extconf.rb
 extra_rdoc_files: []
 files:
-- ".gitignore"
+- .gitignore
 - ChangeLog
 - Gemfile
-- Gemfile.lock
 - LICENSE
 - NOTICE
 - README.md
@@ -463,25 +469,32 @@ files:
 homepage: https://github.com/treasure-data/prestogres
 licenses:
 - Apache 2.0
-metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
   requirements:
-  - - ">="
+  - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: -1048444669451031740
 required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
   requirements:
-  - - ">="
+  - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: -1048444669451031740
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.2.0
+rubygems_version: 1.8.23
 signing_key: 
-specification_version: 4
+specification_version: 3
 summary: Presto PostgreSQL protocol gateway
 test_files: []

checksums.yaml

@@ -1,7 +0,0 @@
----
-SHA1:
-  metadata.gz: fdbbafba90e2dde6d7340746b30d369eda073ed9
-  data.tar.gz: 57d91cbb2bb74c2ce53132ea9309eb9a0cb47e09
-SHA512:
-  metadata.gz: f166e0435a473ec6930522f0a139733f24227a8b0045725fc1e89f4afda25c6fd589c2dc121a46e99c82f79bba135c601b42fec45651c66c185a0aaae6f52eb6
-  data.tar.gz: c351d1a8fe7c59cc7b9a2ec603935766141958df839d4f7179a9921e9410b2da513f6b05beab2f6b5d3d96be309542df876b01fce42f4021f0692d40ceb84652

data/Gemfile.lock

@@ -1,20 +0,0 @@
-PATH
-  remote: .
-  specs:
-    prestogres (0.1.0)
-
-GEM
-  remote: https://rubygems.org/
-  specs:
-    rake (0.9.6)
-    rake-compiler (0.8.3)
-      rake
-
-PLATFORMS
-  ruby
-
-DEPENDENCIES
-  bundler (~> 1.0)
-  prestogres!
-  rake (~> 0.9.2)
-  rake-compiler (~> 0.8.3)