RubyGems - dwh - Versions diffs - 0.1.0 - Mend

dwh 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (104) hide show

checksums.yaml +7 -0
data/.rubocop.yml +36 -0
data/CHANGELOG.md +5 -0
data/LICENSE +21 -0
data/README.md +130 -0
data/Rakefile +42 -0
data/docs/DWH/Adapters/Adapter.html +3053 -0
data/docs/DWH/Adapters/Athena.html +1704 -0
data/docs/DWH/Adapters/Boolean.html +121 -0
data/docs/DWH/Adapters/Druid.html +1626 -0
data/docs/DWH/Adapters/DuckDb.html +2012 -0
data/docs/DWH/Adapters/MySql.html +1704 -0
data/docs/DWH/Adapters/OpenAuthorizable/ClassMethods.html +265 -0
data/docs/DWH/Adapters/OpenAuthorizable.html +1102 -0
data/docs/DWH/Adapters/Postgres.html +2000 -0
data/docs/DWH/Adapters/Snowflake.html +1662 -0
data/docs/DWH/Adapters/SqlServer.html +2084 -0
data/docs/DWH/Adapters/Trino.html +1835 -0
data/docs/DWH/Adapters.html +129 -0
data/docs/DWH/AuthenticationError.html +142 -0
data/docs/DWH/Behaviors.html +767 -0
data/docs/DWH/Capabilities.html +748 -0
data/docs/DWH/Column.html +1115 -0
data/docs/DWH/ConfigError.html +143 -0
data/docs/DWH/ConnectionError.html +143 -0
data/docs/DWH/DWHError.html +138 -0
data/docs/DWH/ExecutionError.html +143 -0
data/docs/DWH/Factory.html +1133 -0
data/docs/DWH/Functions/Arrays.html +505 -0
data/docs/DWH/Functions/Dates.html +1644 -0
data/docs/DWH/Functions/ExtractDatePart.html +804 -0
data/docs/DWH/Functions/Nulls.html +377 -0
data/docs/DWH/Functions.html +846 -0
data/docs/DWH/Logger.html +258 -0
data/docs/DWH/OAuthError.html +138 -0
data/docs/DWH/Settings.html +658 -0
data/docs/DWH/StreamingStats.html +804 -0
data/docs/DWH/Table.html +1260 -0
data/docs/DWH/TableStats.html +583 -0
data/docs/DWH/TokenExpiredError.html +142 -0
data/docs/DWH/UnsupportedCapability.html +135 -0
data/docs/DWH.html +220 -0
data/docs/_index.html +471 -0
data/docs/class_list.html +54 -0
data/docs/css/common.css +1 -0
data/docs/css/full_list.css +58 -0
data/docs/css/style.css +503 -0
data/docs/file.README.html +210 -0
data/docs/file.adapters.html +514 -0
data/docs/file.creating-adapters.html +497 -0
data/docs/file.getting-started.html +288 -0
data/docs/file.usage.html +446 -0
data/docs/file_list.html +79 -0
data/docs/frames.html +22 -0
data/docs/guides/adapters.md +445 -0
data/docs/guides/creating-adapters.md +430 -0
data/docs/guides/getting-started.md +225 -0
data/docs/guides/usage.md +378 -0
data/docs/index.html +210 -0
data/docs/js/app.js +344 -0
data/docs/js/full_list.js +242 -0
data/docs/js/jquery.js +4 -0
data/docs/method_list.html +2038 -0
data/docs/top-level-namespace.html +110 -0
data/lib/dwh/adapters/athena.rb +359 -0
data/lib/dwh/adapters/druid.rb +267 -0
data/lib/dwh/adapters/duck_db.rb +235 -0
data/lib/dwh/adapters/my_sql.rb +235 -0
data/lib/dwh/adapters/open_authorizable.rb +215 -0
data/lib/dwh/adapters/postgres.rb +250 -0
data/lib/dwh/adapters/snowflake.rb +489 -0
data/lib/dwh/adapters/sql_server.rb +257 -0
data/lib/dwh/adapters/trino.rb +213 -0
data/lib/dwh/adapters.rb +363 -0
data/lib/dwh/behaviors.rb +67 -0
data/lib/dwh/capabilities.rb +39 -0
data/lib/dwh/column.rb +79 -0
data/lib/dwh/errors.rb +29 -0
data/lib/dwh/factory.rb +125 -0
data/lib/dwh/functions/arrays.rb +42 -0
data/lib/dwh/functions/dates.rb +162 -0
data/lib/dwh/functions/extract_date_part.rb +70 -0
data/lib/dwh/functions/nulls.rb +31 -0
data/lib/dwh/functions.rb +86 -0
data/lib/dwh/logger.rb +50 -0
data/lib/dwh/settings/athena.yml +77 -0
data/lib/dwh/settings/base.yml +81 -0
data/lib/dwh/settings/databricks.yml +51 -0
data/lib/dwh/settings/druid.yml +59 -0
data/lib/dwh/settings/duckdb.yml +44 -0
data/lib/dwh/settings/mysql.yml +67 -0
data/lib/dwh/settings/postgres.yml +30 -0
data/lib/dwh/settings/redshift.yml +52 -0
data/lib/dwh/settings/snowflake.yml +45 -0
data/lib/dwh/settings/sqlserver.yml +80 -0
data/lib/dwh/settings/trino.yml +77 -0
data/lib/dwh/settings.rb +79 -0
data/lib/dwh/streaming_stats.rb +69 -0
data/lib/dwh/table.rb +105 -0
data/lib/dwh/table_stats.rb +51 -0
data/lib/dwh/version.rb +5 -0
data/lib/dwh.rb +54 -0
data/sig/dwh.rbs +4 -0
metadata +231 -0

data/docs/file.creating-adapters.html ADDED Viewed

@@ -0,0 +1,497 @@
+<!DOCTYPE html>
+<html>
+  <head>
+    <meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>
+  File: Creating Custom Adapters
+    &mdash; Documentation by YARD 0.9.37
+</title>
+  <link rel="stylesheet" href="css/style.css" type="text/css" />
+  <link rel="stylesheet" href="css/common.css" type="text/css" />
+<script type="text/javascript">
+  pathId = "creating-adapters";
+  relpath = '';
+</script>
+  <script type="text/javascript" charset="utf-8" src="js/jquery.js"></script>
+  <script type="text/javascript" charset="utf-8" src="js/app.js"></script>
+  </head>
+  <body>
+    <div class="nav_wrap">
+      <iframe id="nav" src="file_list.html?1"></iframe>
+      <div id="resizer"></div>
+    </div>
+    <div id="main" tabindex="-1">
+      <div id="header">
+        <div id="menu">
+    <a href="_index.html">Index</a> &raquo;
+    <span class="title">File: Creating Custom Adapters</span>
+</div>
+        <div id="search">
+    <a class="full_list_link" id="class_list_link"
+        href="class_list.html">
+        <svg width="24" height="24">
+          <rect x="0" y="4" width="24" height="4" rx="1" ry="1"></rect>
+          <rect x="0" y="12" width="24" height="4" rx="1" ry="1"></rect>
+          <rect x="0" y="20" width="24" height="4" rx="1" ry="1"></rect>
+        </svg>
+    </a>
+</div>
+        <div class="clear"></div>
+      </div>
+      <div id="content"><div id='filecontents'><h1 id="creating-custom-adapters">Creating Custom Adapters</h1>
+<p>The whole point of this library is to make adding a new database integration easy.  With a few steps you can create your own adapter.  If its generic, please contribute back to the project via PR.</p>
+<p>This guide walks you through creating your own custom database adapter for DWH. Creating a new adapter involves extending the base adapter class, implementing required methods, and optionally creating custom settings.</p>
+<h2 id="understanding-dwh-architecture">Understanding DWH Architecture</h2>
+<p>DWH adapters have a simple, focused architecture:</p>
+<ul>
+  <li><strong>5 Core Methods</strong>: Every adapter must implement 5 essential methods</li>
+  <li><strong>YAML Settings</strong>: Database-specific behavior controlled by YAML configuration</li>
+  <li><strong>Configuration Validation</strong>: Automatic validation of connection parameters</li>
+  <li><strong>Function Translation</strong>: SQL functions automatically translated to database-specific syntax</li>
+</ul>
+<h2 id="minimal-adapter-example">Minimal Adapter Example</h2>
+<p>Here’s a minimal adapter implementation:</p>
+<p>```ruby
+module DWH
+  module Adapters
+    class MyCustomAdapter &lt; Adapter
+      # Define required configuration parameters
+      config :host, String, required: true, message: ‘server host ip address or domain name’
+      config :port, Integer, required: false, default: 1234, message: ‘port to connect to’
+      config :database, String, required: true, message: ‘name of database to connect to’
+      config :username, String, required: true, message: ‘connection username’
+      config :password, String, required: false, default: nil, message: ‘connection password’</p>
+<pre class="code ruby"><code class="ruby">  # Implement required methods
+  def connection
+    # Return your database connection object
+    # This is cached, so implement connection reuse here
+    @connection ||= create_connection
+  end
+  def tables(catalog: nil, schema: nil)
+    # Return array of DWH::Table objects
+    # Use catalog/schema for filtering if supported
+  end
+  def metadata(table_name, catalog: nil, schema: nil)
+    # Return single DWH::Table object with column information
+  end
+  def stats(table_name, date_column: nil, catalog: nil, schema: nil)
+    # Return DWH::TableStats object with row counts and date ranges
+  end
+  def execute(sql, format: :array, retries: 0)
+    # Execute SQL and return results in specified format
+    # Formats: :array, :object, :csv, :native
+  end
+  def execute_stream(sql, io, stats: nil)
+    # Execute SQL and stream results directly to IO object
+  end
+  def stream(sql, &amp;block)
+    # Execute SQL and yield chunks to block
+  end
+  private
+  def create_connection
+    # Your database-specific connection logic
+    MyDatabaseClient.connect(
+      host: config[:host],
+      port: config[:port],
+      database: config[:database],
+      username: config[:username],
+      password: config[:password]
+    )
+  end
+end   end end
+</code></pre>
+<h1 id="register-your-adapter">Register your adapter</h1>
+<p>DWH.register(:mycustom, DWH::Adapters::MyCustomAdapter)
+```</p>
+<h2 id="step-by-step-implementation">Step-by-Step Implementation</h2>
+<h3 id="define-configuration-parameters">1. Define Configuration Parameters</h3>
+<p>Use the <code>config</code> class method to define connection parameters:</p>
+<p>```ruby
+class MyCustomAdapter &lt; Adapter
+  # Required parameters
+  config :host, String, required: true, message: ‘server host ip address or domain name’
+  config :database, String, required: true, message: ‘name of database to connect to’</p>
+<p># Optional parameters with defaults
+  config :port, Integer, required: false, default: 5432, message: ‘port to connect to’
+  config :timeout, Integer, required: false, default: 30, message: ‘connection timeout’</p>
+<p># Boolean parameters
+  config :ssl, Boolean, required: false, default: false, message: ‘use ssl connection’</p>
+<p># Parameters with allowed values
+  config :auth_type, String, required: false, default: ‘basic’,
+         message: ‘authentication type’, allowed: %w[basic oauth token]
+end
+```</p>
+<h3 id="implement-connection-management">2. Implement Connection Management</h3>
+<p>```ruby
+def connection
+  return @connection if @connection &amp;&amp; connection_valid?</p>
+<p>@connection = create_connection
+end</p>
+<p>private</p>
+<p>def create_connection
+  # Example for HTTP-based database
+  Faraday.new(
+    url: “#protocol://#config[:host]:#config[:port]”,
+    headers: build_headers,
+    request: {
+      timeout: config[:timeout]
+    }
+  )
+end</p>
+<p>def build_headers
+  headers = { ‘Content-Type’ =&gt; ‘application/json’ }
+  headers[‘Authorization’] = “Bearer #config[:token]” if config[:token]
+  headers
+end</p>
+<p>def connection_valid?
+  # Implement connection health check
+  @connection&amp;.get(‘/health’)&amp;.success?
+rescue
+  false
+end
+```</p>
+<h3 id="implement-table-discovery">3. Implement Table Discovery</h3>
+<p>```ruby
+def tables(catalog: nil, schema: nil)
+  query = build_tables_query(catalog: catalog, schema: schema)
+  results = execute(query, format: :array)</p>
+<p>results.map do |row|
+    DWH::Table.new(
+      physical_name: row[0],
+      schema: row[1] || ‘default’,
+      catalog: row[2],
+      table_type: row[3] || ‘TABLE’
+    )
+  end
+end</p>
+<p>private</p>
+<p>def build_tables_query(catalog: nil, schema: nil)
+  query = “SHOW TABLES”</p>
+<p>conditions = []
+  conditions « “FROM #catalog” if catalog
+  conditions « “LIKE ‘#schema.%’” if schema</p>
+<p>query += “ #‘)” unless conditions.empty?
+  query
+end
+```</p>
+<h3 id="implement-metadata-extraction">4. Implement Metadata Extraction</h3>
+<p>```ruby
+def metadata(table_name, catalog: nil, schema: nil)
+  # Parse table name if it includes schema/catalog
+  parsed = parse_table_name(table_name, catalog: catalog, schema: schema)</p>
+<p>query = build_describe_query(parsed[:table], parsed[:schema], parsed[:catalog])
+  results = execute(query, format: :array)</p>
+<p>columns = results.map do |row|
+    DWH::Column.new(
+      name: row[0],
+      data_type: row[1],
+      normalized_data_type: normalize_data_type(row[1]),
+      nullable: row[2] != ‘NO’,
+      default_value: row[3],
+      character_maximum_length: row[4],
+      numeric_precision: row[5],
+      numeric_scale: row[6]
+    )
+  end</p>
+<p>DWH::Table.new(
+    physical_name: parsed[:table],
+    schema: parsed[:schema],
+    catalog: parsed[:catalog],
+    columns: columns
+  )
+end
+```</p>
+<h3 id="implement-statistics-collection">5. Implement Statistics Collection</h3>
+<p>```ruby
+def stats(table_name, date_column: nil, catalog: nil, schema: nil)
+  parsed = parse_table_name(table_name, catalog: catalog, schema: schema)
+  full_table_name = build_full_table_name(parsed)</p>
+<p># Get row count
+  count_query = “SELECT COUNT(*) FROM #full_table_name”
+  row_count = execute(count_query, format: :array).first.first</p>
+<p># Get date range if date column provided
+  date_start = date_end = nil
+  if date_column
+    date_query = “SELECT MIN(#date_column), MAX(#date_column) FROM #full_table_name”
+    date_result = execute(date_query, format: :array).first
+    date_start, date_end = date_result
+  end</p>
+<p>DWH::TableStats.new(
+    row_count: row_count,
+    date_start: date_start,
+    date_end: date_end
+  )
+end
+```</p>
+<h3 id="implement-query-execution">6. Implement Query Execution</h3>
+<p>```ruby
+def execute(sql, format: :array, retries: 0)
+  response = connection.post(‘/query’, { sql: sql }.to_json)</p>
+<p>raise DWH::ExecutionError, “Query failed: #responseresponse.body” unless response.success?</p>
+<p>raw_data = JSON.parse(response.body)
+  format_results(raw_data, format)
+rescue =&gt; e
+  if retries &gt; 0
+    sleep(1)
+    execute(sql, format: format, retries: retries - 1)
+  else
+    raise DWH::ExecutionError, “Query execution failed: #ee.message”
+  end
+end</p>
+<p>def execute_stream(sql, io, stats: nil)
+  # For HTTP APIs, you might need to paginate or use streaming endpoints
+  offset = 0
+  limit = 10_000</p>
+<p>loop do
+    paginated_sql = “#sql LIMIT #limit OFFSET #offset”
+    results = execute(paginated_sql, format: :array)</p>
+<pre class="code ruby"><code class="ruby">break if results.empty?
+results.each do |row|
+  csv_row = CSV.generate_line(row)
+  io.write(csv_row)
+  stats&amp;.add_row(row)
+end
+offset += limit   end end
+</code></pre>
+<p>def stream(sql, &amp;block)
+  # Similar to execute_stream but yields chunks to block
+  offset = 0
+  limit = 10_000</p>
+<p>loop do
+    paginated_sql = “#sql LIMIT #limit OFFSET #offset”
+    chunk = execute(paginated_sql, format: :array)</p>
+<pre class="code ruby"><code class="ruby">break if chunk.empty?
+yield chunk
+offset += limit   end end
+</code></pre>
+<p>private</p>
+<p>def format_results(raw_data, format)
+  case format
+  when :array
+    raw_data[‘rows’]
+  when :object
+    columns = raw_data[‘columns’]
+    raw_data[‘rows’].map { |row| columns.zip(row).to_h }
+  when :csv
+    CSV.generate do |csv|
+      raw_data[‘rows’].each { |row| csv « row }
+    end
+  when :native
+    raw_data
+  else
+    raise ArgumentError, “Unsupported format: #format”
+  end
+end
+```</p>
+<h2 id="creating-custom-settings">Creating Custom Settings</h2>
+<h3 id="create-settings-file">1. Create Settings File</h3>
+<p>Create by copying the <a href="https://github.com/stratasite/dwh/blob/main/lib/dwh/settings/base.yml">base settings file</a> to a relative directory like so:<code>settings/mycustom.yml</code></p>
+<p>```yaml
+# Override base settings for your database</p>
+<h1 id="function-mappings">Function mappings</h1>
+<p>truncate_date: “DATE_TRUNC(‘@unit’, @exp)”
+date_literal: “DATE(‘@val’)”
+cast: “CAST(@exp AS @type)”</p>
+<h1 id="string-functions">String functions</h1>
+<p>trim: “LTRIM(RTRIM(@exp))”
+upper_case: “UPPER(@exp)”
+lower_case: “LOWER(@exp)”</p>
+<h1 id="null-handling">Null handling</h1>
+<p>if_null: “ISNULL(@exp, @when_null)”
+null_if: “CASE WHEN @exp = @target THEN NULL ELSE @exp END”</p>
+<h1 id="capabilities">Capabilities</h1>
+<p>supports_window_functions: true
+supports_array_functions: false
+supports_common_table_expressions: true
+supports_temp_tables: false</p>
+<h1 id="query-behavior">Query behavior</h1>
+<p>temp_table_type: “subquery”  # options: cte, subquery, temp
+final_pass_measure_join_type: “inner”  # inner, left, right, full</p>
+<h1 id="custom-settings-for-your-database">Custom settings for your database</h1>
+<p>custom_query_prefix: “/* Generated by DWH */”
+max_query_length: 1000000
+```</p>
+<h3 id="custom-settings-location">2. Custom Settings Location</h3>
+<p>```ruby
+class MyCustomAdapter &lt; Adapter
+  # Specify custom settings file location
+  settings_file_path “/path/to/my_custom_settings.yml”</p>
+<p># … rest of implementation
+end
+```</p>
+<h2 id="advanced-features">Advanced Features</h2>
+<h3 id="error-handling">Error Handling</h3>
+<p><code>ruby
+def execute(sql, format: :array, retries: 0)
+  # Your execution logic
+rescue MyDatabaseClient::ConnectionError =&gt; e
+  raise DWH::ConnectionError, "Database connection failed: #{e.message}"
+rescue MyDatabaseClient::QueryError =&gt; e
+  raise DWH::ExecutionError, "Query execution failed: #{e.message}"
+rescue =&gt; e
+  raise DWH::AdapterError, "Unexpected error: #{e.message}"
+end
+</code></p>
+<h3 id="custom-function-translation">Custom Function Translation</h3>
+<p>```ruby
+def custom_function(expression, param1, param2)
+  # Access settings for function templates
+  template = settings[:custom_function] || “CUSTOM_FUNC(@exp, @p1, @p2)”</p>
+<p>template.gsub(‘@exp’, expression)
+          .gsub(‘@p1’, param1.to_s)
+          .gsub(‘@p2’, param2.to_s)
+end
+```</p>
+<h2 id="registration-and-usage">Registration and Usage</h2>
+<h3 id="register-your-adapter-1">Register Your Adapter</h3>
+<p>```ruby
+# In your gem or application initialization
+require ‘dwh’
+require ‘my_custom_adapter’</p>
+<p>DWH.register(:mycustom, DWH::Adapters::MyCustomAdapter)
+```</p>
+<h3 id="use-your-adapter">Use Your Adapter</h3>
+<p>```ruby
+# Create adapter instance
+adapter = DWH.create(:mycustom, {
+  host: ‘database.example.com’,
+  port: 1234,
+  database: ‘analytics’,
+  username: ‘analyst’,
+  password: ‘secret’
+})</p>
+<h1 id="use-standard-dwh-interface">Use standard DWH interface</h1>
+<p>tables = adapter.tables
+metadata = adapter.metadata(‘users’)
+results = adapter.execute(“SELECT COUNT(*) FROM users”)
+```</p>
+<h2 id="examples-to-study">Examples to Study</h2>
+<p>Look at existing adapters for implementation patterns:</p>
+<ul>
+  <li><strong>PostgreSQL</strong> (<code>lib/dwh/adapters/postgres.rb</code>) - RDBMS with full SQL support</li>
+  <li><strong>Druid</strong> (<code>lib/dwh/adapters/druid.rb</code>) - HTTP API-based adapter</li>
+  <li><strong>DuckDB</strong> (<code>lib/dwh/adapters/duck_db.rb</code>) - Embedded database adapter</li>
+</ul>
+</div></div>
+      <div id="footer">
+  Generated on Fri Aug 22 08:31:21 2025 by
+  <a href="https://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
+  0.9.37 (ruby-3.4.4).
+</div>
+    </div>
+  </body>
+</html>