activerecord-dbt 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (29) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +118 -3
  3. data/lib/active_record/dbt/column/column.rb +2 -13
  4. data/lib/active_record/dbt/column/testable/not_null_testable.rb +7 -1
  5. data/lib/active_record/dbt/config.rb +2 -30
  6. data/lib/active_record/dbt/configuration/i18n_configuration.rb +14 -0
  7. data/lib/active_record/dbt/configuration/source.rb +40 -0
  8. data/lib/active_record/dbt/data_type/mapper.rb +81 -14
  9. data/lib/active_record/dbt/dbt_package/dbterd/column/testable/relationships_meta_relationship_type.rb +2 -3
  10. data/lib/active_record/dbt/factory/model/staging_factory.rb +3 -3
  11. data/lib/active_record/dbt/factory/source_factory.rb +1 -2
  12. data/lib/active_record/dbt/i18n_wrapper/translate.rb +34 -0
  13. data/lib/active_record/dbt/model/staging/yml.rb +7 -3
  14. data/lib/active_record/dbt/seed/enum/base.rb +48 -0
  15. data/lib/active_record/dbt/seed/enum/csv.rb +85 -0
  16. data/lib/active_record/dbt/seed/enum/yml.rb +128 -0
  17. data/lib/active_record/dbt/source/yml.rb +6 -2
  18. data/lib/active_record/dbt/table/yml.rb +1 -4
  19. data/lib/active_record/dbt/version.rb +1 -1
  20. data/lib/generators/active_record/dbt/config/templates/source_config.yml.tt +3 -0
  21. data/lib/generators/active_record/dbt/enum/USAGE +9 -0
  22. data/lib/generators/active_record/dbt/enum/enum_generator.rb +31 -0
  23. data/lib/generators/active_record/dbt/staging_model/staging_model_generator.rb +2 -2
  24. metadata +10 -7
  25. data/lib/active_record/dbt/data_type/dwh_platform/apache_spark.rb +0 -27
  26. data/lib/active_record/dbt/data_type/dwh_platform/bigquery.rb +0 -26
  27. data/lib/active_record/dbt/data_type/dwh_platform/postgre_sql.rb +0 -27
  28. data/lib/active_record/dbt/data_type/dwh_platform/redshift.rb +0 -27
  29. data/lib/active_record/dbt/data_type/dwh_platform/snowflake.rb +0 -27
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c7948bb8cc4c14d07c3fca2855b20957a2871c54dd67149fb178029fa93f6f09
4
- data.tar.gz: c8146bfcd3ffa762f16889a79750807ab1e72986e49f34069ac043fe8c89d5f0
3
+ metadata.gz: affd8c9f9864d11cbaf5c35a759ea603c72784b9eae777987debad89fd8cd509
4
+ data.tar.gz: 6b8ca507c1e413143f3d0a7330451767190b3acb731e70c8b2beb87bff9df64b
5
5
  SHA512:
6
- metadata.gz: a2d3ce1aa15cafbd0bb6f282d5faa64f6319d669a38a8dde20b04af3cccf12a21e16fa22cf6f7b87b245dd094bad296de2dd22bd027db047b2ed845cc7de2b8a
7
- data.tar.gz: 6097b5f0b25324679e012d4a16833afba17514f2ebb82eabe778ae510bc8f283c280dd3d6916b2a3220092c63c7827c56b4b0660f14caa6f1b210a496dff7cdc
6
+ metadata.gz: 2798392f8cdcb454a64957a005d7d16e3eeec7e1648234ba5c598cbacb6b683198493337619c9e0a9f17f67a231f6a3fc6f34658651671de3776c7caf28093c9
7
+ data.tar.gz: d417dfdd69d8a0c41baee03a41a0880ecba05abbbb61c86b80ed1ef48021bcdda16a4e2b43dd414a29d7f56ad9cf92a0e3afe0a693b5ae4aa3a38e0bbeb9f03d
data/README.md CHANGED
@@ -58,6 +58,7 @@ dwh_platform | Specify the data warehouse platform to which dbt connects. The de
58
58
  data_sync_delayed | Indicates whether there is a data delay. If set to `true`, `severity: warn` is applied to the `relationships` test. The default is `false`.
59
59
  logger | The destination for log output. The default is `Logger.new('./log/active_record_dbt.log')`.
60
60
  used_dbt_package_names | An array of `dbt` package names to use.
61
+ locale | I18n locale. The default is `I18n.locale`.
61
62
 
62
63
  List of platforms that can currently be set with `dwh_platform`.
63
64
 
@@ -165,10 +166,10 @@ table_overrides:
165
166
 
166
167
  ##### defaults
167
168
 
168
- Set default values for the `name` and `description` of `tables`.
169
+ Set the default value for the `description`(`logical_name`, `description`) of `tables`.
169
170
 
170
- In `logical_name` and `description` of `table_descriptions`, you can refer to the table name with `{{ table_name }}`.
171
- In the `description` of `columns`, you can refer to the table name with `{{ table_name }}` and the column name with `{{ column_name }}`.
171
+ In the `logical_name` and `description` of `table_descriptions`, you can refer to the table name with `{{ table_name }}`.
172
+ In the `description` of `table_descriptions.columns`, you can refer to the table name with `{{ table_name }}` and the column name with `{{ column_name }}`.
172
173
 
173
174
  Example:
174
175
 
@@ -831,6 +832,10 @@ from final
831
832
 
832
833
  Example:
833
834
 
835
+ > [!NOTE]
836
+ >
837
+ > The output will be as shown below. It is recommended to indent the YAML file with a tool of your choice.
838
+
834
839
  ```yaml
835
840
  ---
836
841
  version: 2
@@ -883,6 +888,116 @@ models:
883
888
 
884
889
  ```
885
890
 
891
+ ### Generated dbt Seed Files
892
+
893
+ #### dbt Seed Configuration
894
+
895
+ In the `#{config_directory_path}/source_config.yml` file, describe the properties you want to set for the seed enum.
896
+ You can configure `defaults` in this file.
897
+
898
+ ##### defaults
899
+
900
+ Set the default value for the `description` of the `seeds` enum.
901
+
902
+ In the `description` of `seed_descriptions.enum`, you can refer to the source name with `{{ source_name }}`, the translated table name with `{{ translated_table_name }}`, and the translated column name with `{{ translated_attribute_name }}`.
903
+
904
+ Example:
905
+
906
+ ```yml
907
+ defaults:
908
+ seed_descriptions:
909
+ enum:
910
+ description: "{{ source_name }} {{ translated_table_name }} {{ translated_attribute_name }} enum"
911
+
912
+ ```
913
+
914
+ If nothing is set, it defaults to the following:
915
+
916
+ ```yml
917
+ defaults:
918
+ seed_descriptions:
919
+ enum:
920
+ description: "{{ source_name }} {{ translated_table_name }} {{ translated_attribute_name }} enum"
921
+
922
+ ```
923
+
924
+ #### Generate dbt Seed Enum Files
925
+
926
+ Generate seed enum files for dbt:
927
+
928
+ ```bash
929
+ $ bin/rails generate active_record:dbt:enum TABLE_NAME ENUM_COLUMN_NAME
930
+ ```
931
+
932
+ Generate seed enum files for dbt from the specified `TABLE_NAME` and `ENUM_COLUMN_NAME`.
933
+
934
+ File | Description
935
+ --------- | ---------
936
+ `#{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.csv` | Seed enum file for dbt.
937
+ `#{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.yml` | Seed enum documentation file for dbt.
938
+
939
+ Example:
940
+
941
+ ```bash
942
+ $ bin/rails generate active_record:dbt:enum posts status
943
+ ```
944
+
945
+ ##### Generate `#{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.csv`
946
+
947
+ Example:
948
+
949
+ ```csv
950
+ status_before_type_of_cast,status_key,status_en,status_ja
951
+ 0,draft,Draft,下書き
952
+ 1,published,Published,公開
953
+ 2,deleted,Deleted,削除
954
+
955
+ ```
956
+
957
+ ##### Generate `#{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.yml`
958
+
959
+ Example:
960
+
961
+ > [!NOTE]
962
+ >
963
+ > The output will be as shown below. It is recommended to indent the YAML file with a tool of your choice.
964
+
965
+ ```yaml
966
+ ---
967
+ version: 2
968
+ seeds:
969
+ - name: seed_dummy__post_enum_statuses
970
+ description: dummy Post Status enum
971
+ config:
972
+ column_types:
973
+ status_before_type_of_cast: int64
974
+ status_key: string
975
+ status_en: string
976
+ status_ja: string
977
+ columns:
978
+ - name: status_before_type_of_cast
979
+ description: Status
980
+ tests:
981
+ - unique
982
+ - not_null
983
+ - name: status_key
984
+ description: Status(key)
985
+ tests:
986
+ - unique
987
+ - not_null
988
+ - name: status_en
989
+ description: Status(en)
990
+ tests:
991
+ - unique
992
+ - not_null
993
+ - name: status_ja
994
+ description: Status(ja)
995
+ tests:
996
+ - unique
997
+ - not_null
998
+
999
+ ```
1000
+
886
1001
  ## Contributing
887
1002
 
888
1003
  Contribution directions go here.
@@ -5,6 +5,7 @@ module ActiveRecord
5
5
  module Column
6
6
  class Column
7
7
  include ActiveRecord::Dbt::DataType::Mapper
8
+ include ActiveRecord::Dbt::I18nWrapper::Translate
8
9
 
9
10
  attr_reader :table_name, :column, :column_test, :primary_keys
10
11
 
@@ -23,7 +24,7 @@ module ActiveRecord
23
24
  {
24
25
  'name' => column_name,
25
26
  'description' => description,
26
- 'data_type' => data_type,
27
+ 'data_type' => data_type(column.type),
27
28
  **column_overrides.except(:tests),
28
29
  'tests' => column_test.config
29
30
  }.compact
@@ -45,18 +46,6 @@ module ActiveRecord
45
46
  source_config.dig(:table_descriptions, table_name, :columns, column_name)
46
47
  end
47
48
 
48
- def translated_attribute_name
49
- translated_column_name || translated_default_column_name
50
- end
51
-
52
- def translated_column_name
53
- I18n.t("activerecord.attributes.#{table_name.singularize}.#{column_name}", default: nil)
54
- end
55
-
56
- def translated_default_column_name
57
- I18n.t("attributes.#{column_name}", default: nil)
58
- end
59
-
60
49
  def key_column_name
61
50
  column_name if primary_key? || foreign_key?
62
51
  end
@@ -10,7 +10,13 @@ module ActiveRecord
10
10
  define_required_methods :column
11
11
 
12
12
  def not_null_test
13
- column.null == true ? nil : 'not_null'
13
+ null? ? nil : 'not_null'
14
+ end
15
+
16
+ private
17
+
18
+ def null?
19
+ column.null == true
14
20
  end
15
21
  end
16
22
  end
@@ -9,38 +9,10 @@ module ActiveRecord
9
9
 
10
10
  include ActiveRecord::Dbt::Configuration::DataSync
11
11
  include ActiveRecord::Dbt::Configuration::DwhPlatform
12
+ include ActiveRecord::Dbt::Configuration::I18nConfiguration
12
13
  include ActiveRecord::Dbt::Configuration::Logger
13
- include ActiveRecord::Dbt::Configuration::Parser
14
+ include ActiveRecord::Dbt::Configuration::Source
14
15
  include ActiveRecord::Dbt::Configuration::UsedDbtPackage
15
-
16
- DEFAULT_CONFIG_DIRECTORY_PATH = 'lib/dbt'
17
- DEFAULT_EXPORT_DIRECTORY_PATH = 'doc/dbt'
18
-
19
- attr_writer :config_directory_path, :export_directory_path
20
-
21
- def source_config_path
22
- @source_config_path ||= "#{config_directory_path}/source_config.yml"
23
- end
24
-
25
- def source_config
26
- @source_config ||= parse_yaml(source_config_path)
27
- end
28
-
29
- def source_name
30
- @source_name ||= source_config.dig(:sources, :name).tap do |source_name|
31
- raise SourceNameIsNullError, "'sources.name' is required in '#{source_config_path}'." if source_name.nil?
32
- end
33
- end
34
-
35
- def config_directory_path
36
- @config_directory_path ||= DEFAULT_CONFIG_DIRECTORY_PATH
37
- end
38
-
39
- def export_directory_path
40
- @export_directory_path ||= DEFAULT_EXPORT_DIRECTORY_PATH
41
- end
42
-
43
- class SourceNameIsNullError < StandardError; end
44
16
  end
45
17
  end
46
18
  end
@@ -0,0 +1,14 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Configuration
6
+ module I18nConfiguration
7
+ def locale=(locale = I18n.locale)
8
+ I18n.load_path += Dir[Rails.root.join('config/locales/**/*.{rb,yml}')]
9
+ I18n.locale = locale
10
+ end
11
+ end
12
+ end
13
+ end
14
+ end
@@ -0,0 +1,40 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Configuration
6
+ module Source
7
+ include ActiveRecord::Dbt::Configuration::Parser
8
+
9
+ DEFAULT_CONFIG_DIRECTORY_PATH = 'lib/dbt'
10
+ DEFAULT_EXPORT_DIRECTORY_PATH = 'doc/dbt'
11
+
12
+ attr_writer :config_directory_path, :export_directory_path
13
+
14
+ def config_directory_path
15
+ @config_directory_path ||= DEFAULT_CONFIG_DIRECTORY_PATH
16
+ end
17
+
18
+ def export_directory_path
19
+ @export_directory_path ||= DEFAULT_EXPORT_DIRECTORY_PATH
20
+ end
21
+
22
+ def source_config_path
23
+ @source_config_path ||= "#{config_directory_path}/source_config.yml"
24
+ end
25
+
26
+ def source_config
27
+ @source_config ||= parse_yaml(source_config_path)
28
+ end
29
+
30
+ def source_name
31
+ @source_name ||= source_config.dig(:sources, :name).tap do |source_name|
32
+ raise SourceNameIsNullError, "'sources.name' is required in '#{source_config_path}'." if source_name.nil?
33
+ end
34
+ end
35
+
36
+ class SourceNameIsNullError < StandardError; end
37
+ end
38
+ end
39
+ end
40
+ end
@@ -4,31 +4,98 @@ module ActiveRecord
4
4
  module Dbt
5
5
  module DataType
6
6
  module Mapper
7
- include ActiveRecord::Dbt::DataType::DwhPlatform::ApacheSpark
8
- include ActiveRecord::Dbt::DataType::DwhPlatform::Bigquery
9
- include ActiveRecord::Dbt::DataType::DwhPlatform::PostgreSql
10
- include ActiveRecord::Dbt::DataType::DwhPlatform::Redshift
11
- include ActiveRecord::Dbt::DataType::DwhPlatform::Snowflake
12
-
13
7
  extend ActiveRecord::Dbt::RequiredMethods
14
8
 
15
9
  # [Platform-specific data types | dbt Developer Hub](https://docs.getdbt.com/reference/resource-properties/data-types)
16
10
  RUBY_TO_DWH_PLATFORM_TYPE_MAP = {
17
- 'bigquery' => RUBY_TO_BIGQUERY_TYPES,
18
- 'postgres' => RUBY_TO_POSTGRES_TYPES,
19
- 'redshift' => RUBY_TO_REDSHIFT_TYPES,
20
- 'snowflake' => RUBY_TO_SNOWFLAKE_TYPES,
21
- 'spark' => RUBY_TO_SPARK_TYPES
11
+ # [Data types  |  BigQuery  |  Google Cloud](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#data_type_list)
12
+ 'bigquery' => {
13
+ binary: 'bytes',
14
+ boolean: 'bool',
15
+ date: 'date',
16
+ datetime: 'datetime',
17
+ decimal: 'int64',
18
+ float: 'float64',
19
+ integer: 'int64',
20
+ json: 'json',
21
+ string: 'string',
22
+ text: 'string',
23
+ time: 'time'
24
+ },
25
+
26
+ # TODO: I have not tried it. I don't know if this is the correct data_type.
27
+ # [PostgreSQL: Documentation: 16: Chapter 8. Data Types](https://www.postgresql.org/docs/current/datatype.html)
28
+ 'postgres' => {
29
+ binary: 'bytea',
30
+ boolean: 'boolean',
31
+ date: 'date',
32
+ datetime: 'timestamp',
33
+ decimal: 'bigint',
34
+ float: 'double precision',
35
+ integer: 'bigint',
36
+ json: 'json',
37
+ string: 'text',
38
+ text: 'text',
39
+ time: 'time'
40
+ },
41
+
42
+ # TODO: I have not tried it. I don't know if this is the correct data_type.
43
+ # [Data types - Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/c_Supported_data_types.html)
44
+ 'redshift' => {
45
+ binary: 'varbyte',
46
+ boolean: 'bool',
47
+ date: 'date',
48
+ datetime: 'timestamp',
49
+ decimal: 'decimal',
50
+ float: 'double precision',
51
+ integer: 'integer',
52
+ json: 'super',
53
+ string: 'varchar',
54
+ text: 'varchar',
55
+ time: 'time'
56
+ },
57
+
58
+ # TODO: I have not tried it. I don't know if this is the correct data_type.
59
+ # [Summary of data types | Snowflake Documentation](https://docs.snowflake.com/en/sql-reference/intro-summary-data-types)
60
+ 'snowflake' => {
61
+ binary: 'binary',
62
+ boolean: 'boolean',
63
+ date: 'date',
64
+ datetime: 'datetime',
65
+ decimal: 'decimal',
66
+ float: 'float',
67
+ integer: 'integer',
68
+ json: 'variant',
69
+ string: 'varchar',
70
+ text: 'varchar',
71
+ time: 'time'
72
+ },
73
+
74
+ # TODO: I have not tried it. I don't know if this is the correct data_type.
75
+ # [Data Types - Spark 3.5.1 Documentation](https://spark.apache.org/docs/latest/sql-ref-datatypes.html)
76
+ 'spark' => {
77
+ binary: 'binary',
78
+ boolean: 'boolean',
79
+ date: 'date',
80
+ datetime: 'timestamp',
81
+ decimal: 'decimal',
82
+ float: 'float',
83
+ integer: 'integer',
84
+ json: 'string',
85
+ string: 'string',
86
+ text: 'string',
87
+ time: 'string'
88
+ }
22
89
  }.freeze
23
90
 
24
- define_required_methods :column, :@config
91
+ define_required_methods :@config
25
92
 
26
93
  delegate :dwh_platform, to: :@config
27
94
 
28
95
  private
29
96
 
30
- def data_type
31
- RUBY_TO_DWH_PLATFORM_TYPE_MAP[dwh_platform].fetch(column.type, 'unknown')
97
+ def data_type(type)
98
+ RUBY_TO_DWH_PLATFORM_TYPE_MAP[dwh_platform].fetch(type, 'unknown')
32
99
  end
33
100
  end
34
101
  end
@@ -14,13 +14,12 @@ module ActiveRecord
14
14
  delegate :used_dbterd?, :add_log, to: :@config
15
15
 
16
16
  def relationships_meta_relationship_type
17
- return nil unless used_dbterd?
18
- return nil if no_relationship?
17
+ return nil if !used_dbterd? || no_relationship?
19
18
 
20
19
  {
21
20
  'relationship_type' => relationship_type
22
21
  }
23
- rescue NotSpecifiedOrNotInvalidIdError, StandardError => e
22
+ rescue NotSpecifiedOrNotInvalidIdError, NameError => e
24
23
  relationships_meta_relationship_type_with_active_record_dbt_error(e)
25
24
  end
26
25
 
@@ -5,14 +5,14 @@ module ActiveRecord
5
5
  module Factory
6
6
  module Model
7
7
  module StagingFactory
8
- def self.yml_build(table_name)
8
+ def self.build(table_name)
9
9
  table_factory = ActiveRecord::Dbt::Factory::TableFactory.build(table_name)
10
10
  yml = ActiveRecord::Dbt::Model::Staging::Yml.new(table_factory)
11
- struct = Struct.new(:export_path, :yml_dump, keyword_init: true)
11
+ struct = Struct.new(:export_path, :dump, keyword_init: true)
12
12
 
13
13
  struct.new(
14
14
  export_path: yml.export_path,
15
- yml_dump: YAML.dump(yml.config.deep_stringify_keys)
15
+ dump: yml.dump
16
16
  )
17
17
  end
18
18
  end
@@ -6,9 +6,8 @@ module ActiveRecord
6
6
  module SourceFactory
7
7
  def self.build
8
8
  tables_factory = ActiveRecord::Dbt::Factory::TablesFactory.build
9
- config = ActiveRecord::Dbt::Source::Yml.new(tables_factory).config
10
9
 
11
- YAML.dump(config.deep_stringify_keys)
10
+ ActiveRecord::Dbt::Source::Yml.new(tables_factory).dump
12
11
  end
13
12
  end
14
13
  end
@@ -0,0 +1,34 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module I18nWrapper
6
+ module Translate
7
+ extend ActiveRecord::Dbt::RequiredMethods
8
+
9
+ define_required_methods :table_name
10
+
11
+ delegate :singularize, to: :table_name, prefix: true
12
+
13
+ def translated_table_name
14
+ I18n.t("activerecord.models.#{table_name_singularize}", default: nil)
15
+ end
16
+
17
+ def translated_attribute_name
18
+ @translated_attribute_name ||=
19
+ translated_column_name || translated_default_column_name
20
+ end
21
+
22
+ private
23
+
24
+ def translated_column_name
25
+ I18n.t("activerecord.attributes.#{table_name_singularize}.#{column_name}", default: nil)
26
+ end
27
+
28
+ def translated_default_column_name
29
+ I18n.t("attributes.#{column_name}", default: nil)
30
+ end
31
+ end
32
+ end
33
+ end
34
+ end
@@ -21,7 +21,13 @@ module ActiveRecord
21
21
  "#{basename}.yml"
22
22
  end
23
23
 
24
- def config
24
+ def dump
25
+ YAML.dump(model_config.deep_stringify_keys)
26
+ end
27
+
28
+ private
29
+
30
+ def model_config
25
31
  {
26
32
  'version' => 2,
27
33
  'models' => [
@@ -34,8 +40,6 @@ module ActiveRecord
34
40
  }
35
41
  end
36
42
 
37
- private
38
-
39
43
  def columns
40
44
  @columns ||= sort_columns(table.config['columns'])
41
45
  end
@@ -0,0 +1,48 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Seed
6
+ module Enum
7
+ module Base
8
+ attr_reader :table_name, :enum_column_name
9
+
10
+ delegate :source_name, :export_directory_path, to: :@config
11
+ delegate :singularize, to: :table_name, prefix: true
12
+
13
+ def initialize(table_name, enum_column_name)
14
+ @table_name = table_name
15
+ @enum_column_name = enum_column_name
16
+ @config = ActiveRecord::Dbt::Config.instance
17
+ end
18
+
19
+ private
20
+
21
+ def basename
22
+ "#{export_directory_path}/#{seed_name}"
23
+ end
24
+
25
+ def seed_name
26
+ "seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}"
27
+ end
28
+
29
+ def locales
30
+ @locales ||= I18n.available_locales
31
+ end
32
+
33
+ def application_record_klass
34
+ @application_record_klass ||= table_name_singularize.classify.constantize
35
+ rescue NameError => _e
36
+ raise DoesNotExistTableError, "#{table_name} table does not exist."
37
+ end
38
+
39
+ def enum_pluralized
40
+ enum_column_name.pluralize
41
+ end
42
+
43
+ class DoesNotExistTableError < StandardError; end
44
+ end
45
+ end
46
+ end
47
+ end
48
+ end
@@ -0,0 +1,85 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'csv'
4
+
5
+ module ActiveRecord
6
+ module Dbt
7
+ module Seed
8
+ module Enum
9
+ class Csv
10
+ include ActiveRecord::Dbt::Seed::Enum::Base
11
+
12
+ delegate :singularize, to: :table_name, prefix: true
13
+
14
+ def export_path
15
+ "#{basename}.csv"
16
+ end
17
+
18
+ def dump
19
+ CSV.generate(headers: true) do |csv|
20
+ rows.each { |row| csv << row }
21
+ end
22
+ end
23
+
24
+ private
25
+
26
+ def rows
27
+ [
28
+ header,
29
+ *enum_rows
30
+ ]
31
+ end
32
+
33
+ def header
34
+ [
35
+ "#{enum_column_name}_before_type_of_cast",
36
+ "#{enum_column_name}_key",
37
+ *locale_header
38
+ ]
39
+ end
40
+
41
+ def locale_header
42
+ locales.map { |locale| "#{enum_column_name}_#{locale}" }
43
+ end
44
+
45
+ def enum_rows
46
+ enums.map do |enum_key, enum_before_type_of_cast|
47
+ [
48
+ enum_before_type_of_cast,
49
+ enum_key,
50
+ *enum_values(enum_key)
51
+ ]
52
+ end
53
+ end
54
+
55
+ def enums
56
+ application_record_klass.defined_enums.fetch(enum_column_name)
57
+ rescue KeyError => _e
58
+ raise NotEnumColumnError, "#{table_name}.#{enum_column_name} column is not an enum." if enum_column?
59
+
60
+ raise DoesNotExistColumnError, "The #{enum_column_name} column in the #{table_name} table does not exist."
61
+ end
62
+
63
+ def enum_column?
64
+ application_record_klass.column_names.include?(enum_column_name)
65
+ end
66
+
67
+ def enum_values(enum_key)
68
+ locales.map { |locale| translated_enum_value(enum_key, locale) }
69
+ end
70
+
71
+ def translated_enum_value(enum_key, locale)
72
+ I18n.t(
73
+ "activerecord.enum.#{table_name_singularize}.#{enum_column_name}.#{enum_key}",
74
+ locale: locale,
75
+ default: nil
76
+ )
77
+ end
78
+
79
+ class DoesNotExistColumnError < StandardError; end
80
+ class NotEnumColumnError < StandardError; end
81
+ end
82
+ end
83
+ end
84
+ end
85
+ end
@@ -0,0 +1,128 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Seed
6
+ module Enum
7
+ class Yml
8
+ include ActiveRecord::Dbt::Column::Testable::UniqueTestable
9
+ include ActiveRecord::Dbt::Column::Testable::NotNullTestable
10
+ include ActiveRecord::Dbt::DataType::Mapper
11
+ include ActiveRecord::Dbt::I18nWrapper::Translate
12
+ include ActiveRecord::Dbt::Seed::Enum::Base
13
+
14
+ delegate :source_config, to: :@config
15
+
16
+ alias column_name enum_column_name
17
+
18
+ def export_path
19
+ "#{basename}.yml"
20
+ end
21
+
22
+ def dump
23
+ YAML.dump(seed_config.deep_stringify_keys)
24
+ end
25
+
26
+ private
27
+
28
+ def seed_config
29
+ {
30
+ 'version' => 2,
31
+ 'seeds' => [
32
+ 'name' => seed_name,
33
+ 'description' => seed_description,
34
+ 'config' => {
35
+ 'column_types' => column_types
36
+ }
37
+ ],
38
+ 'columns' => columns
39
+ }
40
+ end
41
+
42
+ def seed_description
43
+ default_seed_description ||
44
+ "#{source_name} #{translated_table_name} #{translated_attribute_name} enum".strip
45
+ end
46
+
47
+ def default_seed_description
48
+ source_config.dig(:defaults, :seed_descriptions, :enum, :description)
49
+ &.gsub(/{{\s*source_name\s*}}/, source_name)
50
+ &.gsub(/{{\s*translated_table_name\s*}}/, translated_table_name)
51
+ &.gsub(/{{\s*translated_attribute_name\s*}}/, translated_attribute_name)
52
+ end
53
+
54
+ def column_types
55
+ {
56
+ "#{enum_column_name}_before_type_of_cast" => data_type(before_type_of_cast_type),
57
+ "#{enum_column_name}_key" => data_type(:string),
58
+ **enum_column_types
59
+ }
60
+ end
61
+
62
+ def before_type_of_cast_type
63
+ application_record_klass.columns_hash[enum_column_name].type
64
+ end
65
+
66
+ def enum_column_types
67
+ locales.each_with_object({}) do |locale, hash|
68
+ hash["#{enum_column_name}_#{locale}"] = data_type(:string)
69
+ end
70
+ end
71
+
72
+ def columns
73
+ [
74
+ before_type_of_cast_column,
75
+ enum_key_column,
76
+ *enum_columns
77
+ ].compact
78
+ end
79
+
80
+ def before_type_of_cast_column
81
+ {
82
+ 'name' => "#{enum_column_name}_before_type_of_cast",
83
+ 'description' => translated_attribute_name,
84
+ 'tests' => tests
85
+ }.compact
86
+ end
87
+
88
+ def enum_key_column
89
+ {
90
+ 'name' => "#{enum_column_name}_key",
91
+ 'description' => "#{translated_attribute_name}(key)",
92
+ 'tests' => tests
93
+ }.compact
94
+ end
95
+
96
+ def enum_columns
97
+ locales.each_with_object([]) do |locale, array|
98
+ array.push(
99
+ {
100
+ 'name' => "#{enum_column_name}_#{locale}",
101
+ 'description' => "#{translated_attribute_name}(#{locale})",
102
+ 'tests' => tests
103
+ }.compact
104
+ )
105
+ end
106
+ end
107
+
108
+ def tests
109
+ [
110
+ unique_test,
111
+ not_null_test
112
+ ].compact.presence
113
+ end
114
+
115
+ # MEMO: I think all enums are unique.
116
+ def unique?
117
+ true
118
+ end
119
+
120
+ # MEMO: I think all enums are null.
121
+ def null?
122
+ false
123
+ end
124
+ end
125
+ end
126
+ end
127
+ end
128
+ end
@@ -13,6 +13,12 @@ module ActiveRecord
13
13
  @config = ActiveRecord::Dbt::Config.instance
14
14
  end
15
15
 
16
+ def dump
17
+ YAML.dump(config.deep_stringify_keys)
18
+ end
19
+
20
+ private
21
+
16
22
  def config
17
23
  {
18
24
  'version' => 2,
@@ -22,8 +28,6 @@ module ActiveRecord
22
28
  }
23
29
  end
24
30
 
25
- private
26
-
27
31
  def source_properties
28
32
  source_config[:sources]
29
33
  end
@@ -4,6 +4,7 @@ module ActiveRecord
4
4
  module Dbt
5
5
  module Table
6
6
  class Yml
7
+ include ActiveRecord::Dbt::I18nWrapper::Translate
7
8
  include ActiveRecord::Dbt::Table::Base
8
9
 
9
10
  attr_reader :table_test, :columns
@@ -51,10 +52,6 @@ module ActiveRecord
51
52
  "Write a logical_name of the '#{table_name}' table."
52
53
  end
53
54
 
54
- def translated_table_name
55
- I18n.t("activerecord.models.#{table_name.singularize}", default: nil)
56
- end
57
-
58
55
  def default_logical_name
59
56
  source_config.dig(:defaults, :table_descriptions, :logical_name)
60
57
  &.gsub(/{{\s*table_name\s*}}/, table_name)
@@ -2,6 +2,6 @@
2
2
 
3
3
  module ActiveRecord
4
4
  module Dbt
5
- VERSION = '0.2.0'
5
+ VERSION = '0.3.0'
6
6
  end
7
7
  end
@@ -40,6 +40,9 @@ defaults:
40
40
  logical_name: Write a logical_name of the '{{ table_name }}' table.
41
41
  columns:
42
42
  description: Write a description of the '{{ table_name }}.{{ column_name }}' column.
43
+ seed_descriptions:
44
+ enum:
45
+ description: "{{ source_name }} {{ translated_table_name }} {{ translated_attribute_name }} enum"
43
46
 
44
47
  table_descriptions:
45
48
  ar_internal_metadata:
@@ -0,0 +1,9 @@
1
+ Description:
2
+ Generate seed enum files for dbt from the specified TABLE_NAME and ENUM_COLUMN_NAME.
3
+
4
+ Example:
5
+ bin/rails generate active_record:dbt:enum TABLE_NAME ENUM_COLUMN_NAME
6
+
7
+ This will create:
8
+ #{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.csv
9
+ #{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.yml
@@ -0,0 +1,31 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Generators
6
+ class EnumGenerator < Rails::Generators::NamedBase
7
+ source_root File.expand_path('templates', __dir__)
8
+
9
+ argument :enum_column_name, type: :string, default: nil
10
+
11
+ def create_enum_seed_csv_file
12
+ create_file csv.export_path, csv.dump
13
+ end
14
+
15
+ def create_enum_seed_yml_file
16
+ create_file yml.export_path, yml.dump
17
+ end
18
+
19
+ private
20
+
21
+ def csv
22
+ @csv ||= ActiveRecord::Dbt::Seed::Enum::Csv.new(name, enum_column_name)
23
+ end
24
+
25
+ def yml
26
+ @yml ||= ActiveRecord::Dbt::Seed::Enum::Yml.new(name, enum_column_name)
27
+ end
28
+ end
29
+ end
30
+ end
31
+ end
@@ -11,7 +11,7 @@ module ActiveRecord
11
11
  end
12
12
 
13
13
  def create_staging_model_yml_file
14
- create_file yml.export_path, yml.yml_dump
14
+ create_file yml.export_path, yml.dump
15
15
  end
16
16
 
17
17
  private
@@ -21,7 +21,7 @@ module ActiveRecord
21
21
  end
22
22
 
23
23
  def yml
24
- @yml ||= ActiveRecord::Dbt::Factory::Model::StagingFactory.yml_build(name)
24
+ @yml ||= ActiveRecord::Dbt::Factory::Model::StagingFactory.build(name)
25
25
  end
26
26
 
27
27
  def source_paths
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: activerecord-dbt
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - yamotech
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2024-08-03 00:00:00.000000000 Z
11
+ date: 2024-08-14 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activerecord
@@ -73,14 +73,11 @@ files:
73
73
  - lib/active_record/dbt/config.rb
74
74
  - lib/active_record/dbt/configuration/data_sync.rb
75
75
  - lib/active_record/dbt/configuration/dwh_platform.rb
76
+ - lib/active_record/dbt/configuration/i18n_configuration.rb
76
77
  - lib/active_record/dbt/configuration/logger.rb
77
78
  - lib/active_record/dbt/configuration/parser.rb
79
+ - lib/active_record/dbt/configuration/source.rb
78
80
  - lib/active_record/dbt/configuration/used_dbt_package.rb
79
- - lib/active_record/dbt/data_type/dwh_platform/apache_spark.rb
80
- - lib/active_record/dbt/data_type/dwh_platform/bigquery.rb
81
- - lib/active_record/dbt/data_type/dwh_platform/postgre_sql.rb
82
- - lib/active_record/dbt/data_type/dwh_platform/redshift.rb
83
- - lib/active_record/dbt/data_type/dwh_platform/snowflake.rb
84
81
  - lib/active_record/dbt/data_type/mapper.rb
85
82
  - lib/active_record/dbt/dbt_package/dbt_utils/table/testable/unique_combination_of_columns_testable.rb
86
83
  - lib/active_record/dbt/dbt_package/dbterd/column/testable/relationships_meta_relationship_type.rb
@@ -89,11 +86,15 @@ files:
89
86
  - lib/active_record/dbt/factory/source_factory.rb
90
87
  - lib/active_record/dbt/factory/table_factory.rb
91
88
  - lib/active_record/dbt/factory/tables_factory.rb
89
+ - lib/active_record/dbt/i18n_wrapper/translate.rb
92
90
  - lib/active_record/dbt/model/staging/base.rb
93
91
  - lib/active_record/dbt/model/staging/sql.rb
94
92
  - lib/active_record/dbt/model/staging/yml.rb
95
93
  - lib/active_record/dbt/railtie.rb
96
94
  - lib/active_record/dbt/required_methods.rb
95
+ - lib/active_record/dbt/seed/enum/base.rb
96
+ - lib/active_record/dbt/seed/enum/csv.rb
97
+ - lib/active_record/dbt/seed/enum/yml.rb
97
98
  - lib/active_record/dbt/source/yml.rb
98
99
  - lib/active_record/dbt/table/base.rb
99
100
  - lib/active_record/dbt/table/test.rb
@@ -102,6 +103,8 @@ files:
102
103
  - lib/generators/active_record/dbt/config/USAGE
103
104
  - lib/generators/active_record/dbt/config/config_generator.rb
104
105
  - lib/generators/active_record/dbt/config/templates/source_config.yml.tt
106
+ - lib/generators/active_record/dbt/enum/USAGE
107
+ - lib/generators/active_record/dbt/enum/enum_generator.rb
105
108
  - lib/generators/active_record/dbt/initializer/USAGE
106
109
  - lib/generators/active_record/dbt/initializer/initializer_generator.rb
107
110
  - lib/generators/active_record/dbt/initializer/templates/dbt.rb
@@ -1,27 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module ApacheSpark
8
- # TODO: I have not tried it. I don't know if this is the correct data_type.
9
- # [Data Types - Spark 3.5.1 Documentation](https://spark.apache.org/docs/latest/sql-ref-datatypes.html)
10
- RUBY_TO_SPARK_TYPES = {
11
- binary: 'binary',
12
- boolean: 'boolean',
13
- date: 'date',
14
- datetime: 'timestamp',
15
- decimal: 'decimal',
16
- float: 'float',
17
- integer: 'integer',
18
- json: 'string',
19
- string: 'string',
20
- text: 'string',
21
- time: 'string'
22
- }.freeze
23
- end
24
- end
25
- end
26
- end
27
- end
@@ -1,26 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module Bigquery
8
- # [Data types  |  BigQuery  |  Google Cloud](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#data_type_list)
9
- RUBY_TO_BIGQUERY_TYPES = {
10
- binary: 'bytes',
11
- boolean: 'bool',
12
- date: 'date',
13
- datetime: 'datetime',
14
- decimal: 'int64',
15
- float: 'float64',
16
- integer: 'int64',
17
- json: 'json',
18
- string: 'string',
19
- text: 'string',
20
- time: 'time'
21
- }.freeze
22
- end
23
- end
24
- end
25
- end
26
- end
@@ -1,27 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module PostgreSql
8
- # TODO: I have not tried it. I don't know if this is the correct data_type.
9
- # [PostgreSQL: Documentation: 16: Chapter 8. Data Types](https://www.postgresql.org/docs/current/datatype.html)
10
- RUBY_TO_POSTGRES_TYPES = {
11
- binary: 'bytea',
12
- boolean: 'boolean',
13
- date: 'date',
14
- datetime: 'timestamp',
15
- decimal: 'bigint',
16
- float: 'double precision',
17
- integer: 'bigint',
18
- json: 'json',
19
- string: 'text',
20
- text: 'text',
21
- time: 'time'
22
- }.freeze
23
- end
24
- end
25
- end
26
- end
27
- end
@@ -1,27 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module Redshift
8
- # TODO: I have not tried it. I don't know if this is the correct data_type.
9
- # [Data types - Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/c_Supported_data_types.html)
10
- RUBY_TO_REDSHIFT_TYPES = {
11
- binary: 'varbyte',
12
- boolean: 'bool',
13
- date: 'date',
14
- datetime: 'timestamp',
15
- decimal: 'decimal',
16
- float: 'double precision',
17
- integer: 'integer',
18
- json: 'super',
19
- string: 'varchar',
20
- text: 'varchar',
21
- time: 'time'
22
- }.freeze
23
- end
24
- end
25
- end
26
- end
27
- end
@@ -1,27 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module Snowflake
8
- # TODO: I have not tried it. I don't know if this is the correct data_type.
9
- # [Summary of data types | Snowflake Documentation](https://docs.snowflake.com/en/sql-reference/intro-summary-data-types)
10
- RUBY_TO_SNOWFLAKE_TYPES = {
11
- binary: 'binary',
12
- boolean: 'boolean',
13
- date: 'date',
14
- datetime: 'datetime',
15
- decimal: 'decimal',
16
- float: 'float',
17
- integer: 'integer',
18
- json: 'variant',
19
- string: 'varchar',
20
- text: 'varchar',
21
- time: 'time'
22
- }.freeze
23
- end
24
- end
25
- end
26
- end
27
- end