exwiw 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 0aa1977437fc4e44349ecc11431e4f8697acb1471e159777c629850a52de1664
+  data.tar.gz: cd92c7a05f6958d2f0cc5ee1204b30fe7b6bd6e168aa18c0b2b182da3b00af9a
+SHA512:
+  metadata.gz: e979a144ac442f73483c23c93cb335742b43e9d198811788edb9a416a3d70cd3b23b64ab0783cf20b4452afa00d46827aaed629d021d880172d373491cc9ec96
+  data.tar.gz: 35375bb916081981ff264dd74a53b59638c78ad0e2f4dfb40d8716bec59caa387dc114001234be8bb2ba20a3345ecac9d88a1b7b0322ae38b8d92759c61efa75

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## [Unreleased]
+
+## [0.1.0] - 2025-01-31
+
+- Initial release

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Shia
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,183 @@
+# Exwiw
+
+Export What I Want (Exwiw) is a Ruby gem that allows you to export records from a database to a dump file(to specifically, the full list of INSERT sql) on the specified conditions.
+
+## When to use
+
+Most of case in developing a software, There is no better choice than the same data in production.
+You might make well-crafted data, but it's very very hard to maintain.
+
+If you find the way to maintain the data for develoment env, then exwiw might be a solution for that.
+
+- Export the full database and mask data and import to another database.
+- Setup some system to replicate and mask data in real-time to another database.
+
+
+You want to export only the data you want to export.
+
+## Features
+
+- Export the full list of INSERT sql for the specified conditions.
+- Provide serveral masking options for sensitive columns.
+- Provide config generator for ActiveRecord.
+
+## Installation
+
+```bash
+bundle add exwiw
+```
+
+Most of cases, you want to add 'require: false' to the Gemfile.
+
+If bundler is not being used to manage dependencies, install the gem by executing:
+
+```bash
+gem install exwiw
+```
+
+## Supported Databases
+
+- mysql2
+- postgresql
+- sqlite3
+
+## Usage
+
+### Command
+
+```bash
+# dump & masking all records from database to dump.sql based on schema.json
+# pass database password as an environment variable 'DATABASE_PASSWORD'
+exwiw \
+  --adapter=mysql2 \
+  --host=localhost \
+  --port=3306 \
+  --user=reader \
+  --database=app_production \
+  --config-dir=exwiw \
+  --target-table=shops \
+  --ids=1 \ # comma separated ids
+  --output-dir=dump \
+  --log-level=info
+```
+
+This command will generate sql files in the `dump` directory.
+
+- `dump/insert-{idx}-{table_name}.sql`
+- `dump/delete-{idx}-{table_name}.sql`
+
+idx means the order of the dump. bigger idx might depend on smaller idx,
+so you should import the dump in order.
+
+you need to delete the records before importing the dump,
+`delete-{idx}-{table_name}.sql` will help you to do that.
+This sql will delete "all" related records to the extract targets.
+idx meaning is the same as insert sql.
+
+### Generator
+
+the config generator is provided as Rake task.
+
+```bash
+# generate table schema under exwiw/
+bundle exec rake exwiw:schema:generate
+```
+
+### Configuration
+
+This is an example of the one table schema:
+
+```json
+{
+    "name": "users",
+    "primary_key": "id",
+    "filter": "users.id > 0",
+    "belongs_to": [{
+        "name": "companies",
+        "foreign_key": "company_id"
+    }],
+    "columns": [{
+        "name": "id",
+    }, {
+        "name": "email",
+        "replace_with": "user{id}@example.com"
+    }, {
+        "name": "company_id"
+    }]
+}
+```
+
+`--config-dir` will use all json files in the specified directory.
+
+### Filter
+
+Some case, you don't need full records related to target. e.g. dump user access logs only for the last year.
+`filter` is here for that. Be careful to use this option, as it will be:
+
+- injected as it is in table condition(e.g. WHERE on mysql), so you are recommended to clearify table name of column to avoid ambiguity.
+- injected to every where / join clause, so it affects to all tables depends on filterted target-table. it results to data inconsistency.
+
+### Masking
+
+`exwiw` provides several options for masking value.
+
+#### `replace_with`
+
+It will replace the value with the specified string,
+and you can use the column name with `{}` to replace the value with the column value.
+
+For example, Let assume we have the record which id is 1,
+then "user{id}@example.com" will be replaced with "user1@example.com".
+
+#### `raw_sql`
+
+It will used instead of the original value.
+
+For example, `"raw_sql": "CONCAT('user', shops.id, '@example.com')"` is equivalent to
+`"replace_with": "user{id}@example.com"`.
+This is useful when you want to transform with functions provided by the database.
+
+Notice that you are recommended to clearify table name of column to avoid ambiguity.
+
+If it used with `replace_with`, `replace_with` will be ignored.
+
+#### `map`
+
+XXX: TODO
+
+Given value will be evaluated as Ruby code, and treated as the proc.
+
+```
+"map": "proc { |r| 'user' + v['id'].to_s + '@example.com' }"
+```
+
+which is equivalent to `"replace_with": "user{id}@example.com"`.
+
+Notice this is the most powerful option, but you should be careful to use this option.
+Because this transformation occured on exwiw process, so much slower than other options.
+Most of case, this option is not recommended.
+
+## How it works
+
+- Load the table information from the specified config file.
+- Calculate the dependency between tables.
+- Generate the full list of INSERT sql based on the specified conditions.
+  - If the processing table has no relation with target tables, then dump all records.
+  - If the processing table has relation with target tables, then dump the records which are related to the target tables.
+- Generate the full list of DELETE sql based on the specified conditions.
+  - If the processing table has no relation with target tables, then delete all records.
+  - If the processing table has relation with target tables, then delete the records which are related to the target tables.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/riseshia/exwiw.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/exe/exwiw

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'exwiw/cli'
+
+Exwiw::CLI.start(ARGV)

data/lib/exwiw/adapter/mysql2_adapter.rb

@@ -0,0 +1,171 @@
+# frozen_string_literal: true
+
+module Exwiw
+  module Adapter
+    class Mysql2Adapter < Base
+      def execute(query_ast)
+        sql = compile_ast(query_ast)
+
+        @logger.debug("  Executing SQL: \n#{sql}")
+        connection.query(sql, cast: false, as: :array).to_a
+      end
+
+      def to_bulk_insert(results, table)
+        table_name = table.name
+
+        value_list = results.map do |row|
+          quoted_values = row.map do |value|
+            escape_value(value)
+          end
+          "(" + quoted_values.join(', ') + ")"
+        end
+        values = value_list.join(",\n")
+
+        column_names = table.columns.map(&:name).join(', ')
+        "INSERT INTO #{table_name} (#{column_names}) VALUES\n#{values};"
+      end
+
+      def to_bulk_delete(select_query_ast, table)
+        raise NotImplementedError unless select_query_ast.is_a?(Exwiw::QueryAst::Select)
+
+        sql = "DELETE FROM #{select_query_ast.from_table_name}"
+
+        if select_query_ast.join_clauses.empty?
+          # Ignore filter option, because bulk delete is for cleaning before import,
+          # so it should delete all records to avoid foreign key violation & data consistancy.
+          compiled_where_conditions = select_query_ast.
+            where_clauses.
+            select { |where| where.is_a?(Exwiw::QueryAst::WhereClause) }.
+            map do |where|
+            compile_where_condition(where, select_query_ast.from_table_name)
+          end
+
+          if compiled_where_conditions.size > 0
+            sql += "\nWHERE "
+            sql += compiled_where_conditions.join(' AND ')
+          end
+          sql += ";"
+
+          return sql
+        end
+
+        subquery_ast = Exwiw::QueryAst::Select.new
+        first_join = select_query_ast.join_clauses.first.clone
+
+        subquery_ast.from(first_join.join_table_name)
+        primay_key_col = table.columns.find { |col| col.name == table.primary_key }
+        subquery_ast.select([primay_key_col])
+        select_query_ast.join_clauses[1..].each do |join|
+          subquery_ast.join(join)
+        end
+        first_join.where_clauses.each do |where|
+          # Ignore filter option, because bulk delete is for cleaning before import,
+          # so it should delete all records to avoid foreign key violation & data consistancy.
+          subquery_ast.where(where) if where.is_a?(Exwiw::QueryAst::WhereClause)
+        end
+
+        foreign_key = first_join.foreign_key
+        subquery_sql = compile_ast(subquery_ast)
+        sql += "\nWHERE #{select_query_ast.from_table_name}.#{foreign_key} IN (#{subquery_sql});"
+
+        sql
+      end
+
+      def compile_ast(query_ast)
+        raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
+
+        sql = "SELECT "
+        sql += query_ast.columns.map { |col| compile_column_name(query_ast, col) }.join(', ')
+        sql += " FROM #{query_ast.from_table_name}"
+
+        query_ast.join_clauses.each do |join|
+          sql += " JOIN #{join.join_table_name} ON #{query_ast.from_table_name}.#{join.foreign_key} = #{join.join_table_name}.#{join.primary_key}"
+
+          join.where_clauses.each do |where|
+            compiled_where_condition = compile_where_condition(where, join.join_table_name)
+            sql += " AND #{compiled_where_condition}"
+          end
+        end
+
+        if query_ast.where_clauses.any?
+          sql += " WHERE "
+          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+        end
+
+        sql
+      end
+
+      private def compile_where_condition(where_clause, table_name)
+        # Use as it is if it's a raw query
+        return where_clause if where_clause.is_a?(String)
+
+        key = "#{table_name}.#{where_clause.column_name}"
+
+        if where_clause.operator == :eq
+          values = where_clause.value.map { |v| escape_value(v) }
+
+          if values.size == 1
+            "#{key} = #{values.first}"
+          else
+            "#{key} IN (#{values.join(', ')})"
+          end
+        else
+          raise "Unsupported operator: #{where_clause.operator}"
+        end
+      end
+
+      private def escape_value(value)
+        case value
+        when nil
+          "NULL"
+        when String
+          qv = escape_single_quote(value)
+          "'#{qv}'"
+        else
+          value
+        end
+      end
+
+      private def escape_single_quote(value)
+        value.gsub("'", "''")
+      end
+
+      private def compile_column_name(ast, column)
+        case column
+        when Exwiw::QueryAst::ColumnValue::Plain
+          "#{ast.from_table_name}.#{column.name}"
+        when Exwiw::QueryAst::ColumnValue::RawSql
+          column.value
+        when Exwiw::QueryAst::ColumnValue::ReplaceWith
+          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
+            if part.start_with?('{')
+              name = part[1..-2]
+              "#{ast.from_table_name}.#{name}"
+            else
+              "'#{part}'"
+            end
+          end
+
+          replaced = parts.join(", ")
+          "CONCAT(#{replaced})"
+        else
+          raise "Unreachable case: #{column.inspect}"
+        end
+      end
+
+      private def connection
+        @connection ||=
+          begin
+            require 'mysql2'
+            Mysql2::Client.new(
+              host: @connection_config.host,
+              port: @connection_config.port,
+              username: @connection_config.user,
+              password: @connection_config.password,
+              database: @connection_config.database_name
+            )
+          end
+      end
+    end
+  end
+end

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -0,0 +1,171 @@
+# frozen_string_literal: true
+
+module Exwiw
+  module Adapter
+    class PostgresqlAdapter < Base
+      def execute(query_ast)
+        sql = compile_ast(query_ast)
+
+        @logger.debug("  Executing SQL: \n#{sql}")
+        connection.exec(sql).values
+      end
+
+      def to_bulk_insert(results, table)
+        table_name = table.name
+
+        value_list = results.map do |row|
+          quoted_values = row.map do |value|
+            escape_value(value)
+          end
+          "(" + quoted_values.join(', ') + ")"
+        end
+        values = value_list.join(",\n")
+
+        column_names = table.columns.map(&:name).join(', ')
+        "INSERT INTO #{table_name} (#{column_names}) VALUES\n#{values};"
+      end
+
+      def to_bulk_delete(select_query_ast, table)
+        raise NotImplementedError unless select_query_ast.is_a?(Exwiw::QueryAst::Select)
+
+        sql = "DELETE FROM #{select_query_ast.from_table_name}"
+
+        if select_query_ast.join_clauses.empty?
+          # Ignore filter option, because bulk delete is for cleaning before import,
+          # so it should delete all records to avoid foreign key violation & data consistancy.
+          compiled_where_conditions = select_query_ast.
+            where_clauses.
+            select { |where| where.is_a?(Exwiw::QueryAst::WhereClause) }.
+            map do |where|
+            compile_where_condition(where, select_query_ast.from_table_name)
+          end
+
+          if compiled_where_conditions.size > 0
+            sql += "\nWHERE "
+            sql += compiled_where_conditions.join(' AND ')
+          end
+          sql += ";"
+
+          return sql
+        end
+
+        subquery_ast = Exwiw::QueryAst::Select.new
+        first_join = select_query_ast.join_clauses.first.clone
+
+        subquery_ast.from(first_join.join_table_name)
+        primay_key_col = table.columns.find { |col| col.name == table.primary_key }
+        subquery_ast.select([primay_key_col])
+        select_query_ast.join_clauses[1..].each do |join|
+          subquery_ast.join(join)
+        end
+        first_join.where_clauses.each do |where|
+          # Ignore filter option, because bulk delete is for cleaning before import,
+          # so it should delete all records to avoid foreign key violation & data consistancy.
+          subquery_ast.where(where) if where.is_a?(Exwiw::QueryAst::WhereClause)
+        end
+
+        foreign_key = first_join.foreign_key
+        subquery_sql = compile_ast(subquery_ast)
+        sql += "\nWHERE #{select_query_ast.from_table_name}.#{foreign_key} IN (#{subquery_sql});"
+
+        sql
+      end
+
+      def compile_ast(query_ast)
+        raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
+
+        sql = "SELECT "
+        sql += query_ast.columns.map { |col| compile_column_name(query_ast, col) }.join(', ')
+        sql += " FROM #{query_ast.from_table_name}"
+
+        query_ast.join_clauses.each do |join|
+          sql += " JOIN #{join.join_table_name} ON #{query_ast.from_table_name}.#{join.foreign_key} = #{join.join_table_name}.#{join.primary_key}"
+
+          join.where_clauses.each do |where|
+            compiled_where_condition = compile_where_condition(where, join.join_table_name)
+            sql += " AND #{compiled_where_condition}"
+          end
+        end
+
+        if query_ast.where_clauses.any?
+          sql += " WHERE "
+          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+        end
+
+        sql
+      end
+
+      private def compile_where_condition(where_clause, table_name)
+        # Use as it is if it's a raw query
+        return where_clause if where_clause.is_a?(String)
+
+        key = "#{table_name}.#{where_clause.column_name}"
+
+        if where_clause.operator == :eq
+          values = where_clause.value.map { |v| escape_value(v) }
+
+          if values.size == 1
+            "#{key} = #{values.first}"
+          else
+            "#{key} IN (#{values.join(', ')})"
+          end
+        else
+          raise "Unsupported operator: #{where_clause.operator}"
+        end
+      end
+
+      private def escape_value(value)
+        case value
+        when nil
+          "NULL"
+        when String
+          qv = escape_single_quote(value)
+          "'#{qv}'"
+        else
+          value
+        end
+      end
+
+      private def escape_single_quote(value)
+        value.gsub("'", "''")
+      end
+
+      private def compile_column_name(ast, column)
+        case column
+        when Exwiw::QueryAst::ColumnValue::Plain
+          "#{ast.from_table_name}.#{column.name}"
+        when Exwiw::QueryAst::ColumnValue::RawSql
+          column.value
+        when Exwiw::QueryAst::ColumnValue::ReplaceWith
+          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
+            if part.start_with?('{')
+              name = part[1..-2]
+              "#{ast.from_table_name}.#{name}"
+            else
+              "'#{part}'"
+            end
+          end
+
+          replaced = parts.join(", ")
+          "CONCAT(#{replaced})"
+        else
+          raise "Unreachable case: #{column.inspect}"
+        end
+      end
+
+      private def connection
+        @connection ||=
+          begin
+            require 'pg'
+            PG.connect(
+              host: @connection_config.host,
+              port: @connection_config.port,
+              user: @connection_config.user,
+              password: @connection_config.password,
+              dbname: @connection_config.database_name
+            )
+          end
+      end
+    end
+  end
+end