activerecord-dbt 0.2.0 → 0.3.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (29) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +118 -3
  3. data/lib/active_record/dbt/column/column.rb +2 -13
  4. data/lib/active_record/dbt/column/testable/not_null_testable.rb +7 -1
  5. data/lib/active_record/dbt/config.rb +2 -30
  6. data/lib/active_record/dbt/configuration/i18n_configuration.rb +14 -0
  7. data/lib/active_record/dbt/configuration/source.rb +40 -0
  8. data/lib/active_record/dbt/data_type/mapper.rb +81 -14
  9. data/lib/active_record/dbt/dbt_package/dbterd/column/testable/relationships_meta_relationship_type.rb +2 -3
  10. data/lib/active_record/dbt/factory/model/staging_factory.rb +3 -3
  11. data/lib/active_record/dbt/factory/source_factory.rb +1 -2
  12. data/lib/active_record/dbt/i18n_wrapper/translate.rb +34 -0
  13. data/lib/active_record/dbt/model/staging/yml.rb +7 -3
  14. data/lib/active_record/dbt/seed/enum/base.rb +48 -0
  15. data/lib/active_record/dbt/seed/enum/csv.rb +85 -0
  16. data/lib/active_record/dbt/seed/enum/yml.rb +128 -0
  17. data/lib/active_record/dbt/source/yml.rb +6 -2
  18. data/lib/active_record/dbt/table/yml.rb +1 -4
  19. data/lib/active_record/dbt/version.rb +1 -1
  20. data/lib/generators/active_record/dbt/config/templates/source_config.yml.tt +3 -0
  21. data/lib/generators/active_record/dbt/enum/USAGE +9 -0
  22. data/lib/generators/active_record/dbt/enum/enum_generator.rb +31 -0
  23. data/lib/generators/active_record/dbt/staging_model/staging_model_generator.rb +2 -2
  24. metadata +10 -7
  25. data/lib/active_record/dbt/data_type/dwh_platform/apache_spark.rb +0 -27
  26. data/lib/active_record/dbt/data_type/dwh_platform/bigquery.rb +0 -26
  27. data/lib/active_record/dbt/data_type/dwh_platform/postgre_sql.rb +0 -27
  28. data/lib/active_record/dbt/data_type/dwh_platform/redshift.rb +0 -27
  29. data/lib/active_record/dbt/data_type/dwh_platform/snowflake.rb +0 -27
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c7948bb8cc4c14d07c3fca2855b20957a2871c54dd67149fb178029fa93f6f09
4
- data.tar.gz: c8146bfcd3ffa762f16889a79750807ab1e72986e49f34069ac043fe8c89d5f0
3
+ metadata.gz: affd8c9f9864d11cbaf5c35a759ea603c72784b9eae777987debad89fd8cd509
4
+ data.tar.gz: 6b8ca507c1e413143f3d0a7330451767190b3acb731e70c8b2beb87bff9df64b
5
5
  SHA512:
6
- metadata.gz: a2d3ce1aa15cafbd0bb6f282d5faa64f6319d669a38a8dde20b04af3cccf12a21e16fa22cf6f7b87b245dd094bad296de2dd22bd027db047b2ed845cc7de2b8a
7
- data.tar.gz: 6097b5f0b25324679e012d4a16833afba17514f2ebb82eabe778ae510bc8f283c280dd3d6916b2a3220092c63c7827c56b4b0660f14caa6f1b210a496dff7cdc
6
+ metadata.gz: 2798392f8cdcb454a64957a005d7d16e3eeec7e1648234ba5c598cbacb6b683198493337619c9e0a9f17f67a231f6a3fc6f34658651671de3776c7caf28093c9
7
+ data.tar.gz: d417dfdd69d8a0c41baee03a41a0880ecba05abbbb61c86b80ed1ef48021bcdda16a4e2b43dd414a29d7f56ad9cf92a0e3afe0a693b5ae4aa3a38e0bbeb9f03d
data/README.md CHANGED
@@ -58,6 +58,7 @@ dwh_platform | Specify the data warehouse platform to which dbt connects. The de
58
58
  data_sync_delayed | Indicates whether there is a data delay. If set to `true`, `severity: warn` is applied to the `relationships` test. The default is `false`.
59
59
  logger | The destination for log output. The default is `Logger.new('./log/active_record_dbt.log')`.
60
60
  used_dbt_package_names | An array of `dbt` package names to use.
61
+ locale | I18n locale. The default is `I18n.locale`.
61
62
 
62
63
  List of platforms that can currently be set with `dwh_platform`.
63
64
 
@@ -165,10 +166,10 @@ table_overrides:
165
166
 
166
167
  ##### defaults
167
168
 
168
- Set default values for the `name` and `description` of `tables`.
169
+ Set the default value for the `description`(`logical_name`, `description`) of `tables`.
169
170
 
170
- In `logical_name` and `description` of `table_descriptions`, you can refer to the table name with `{{ table_name }}`.
171
- In the `description` of `columns`, you can refer to the table name with `{{ table_name }}` and the column name with `{{ column_name }}`.
171
+ In the `logical_name` and `description` of `table_descriptions`, you can refer to the table name with `{{ table_name }}`.
172
+ In the `description` of `table_descriptions.columns`, you can refer to the table name with `{{ table_name }}` and the column name with `{{ column_name }}`.
172
173
 
173
174
  Example:
174
175
 
@@ -831,6 +832,10 @@ from final
831
832
 
832
833
  Example:
833
834
 
835
+ > [!NOTE]
836
+ >
837
+ > The output will be as shown below. It is recommended to indent the YAML file with a tool of your choice.
838
+
834
839
  ```yaml
835
840
  ---
836
841
  version: 2
@@ -883,6 +888,116 @@ models:
883
888
 
884
889
  ```
885
890
 
891
+ ### Generated dbt Seed Files
892
+
893
+ #### dbt Seed Configuration
894
+
895
+ In the `#{config_directory_path}/source_config.yml` file, describe the properties you want to set for the seed enum.
896
+ You can configure `defaults` in this file.
897
+
898
+ ##### defaults
899
+
900
+ Set the default value for the `description` of the `seeds` enum.
901
+
902
+ In the `description` of `seed_descriptions.enum`, you can refer to the source name with `{{ source_name }}`, the translated table name with `{{ translated_table_name }}`, and the translated column name with `{{ translated_attribute_name }}`.
903
+
904
+ Example:
905
+
906
+ ```yml
907
+ defaults:
908
+ seed_descriptions:
909
+ enum:
910
+ description: "{{ source_name }} {{ translated_table_name }} {{ translated_attribute_name }} enum"
911
+
912
+ ```
913
+
914
+ If nothing is set, it defaults to the following:
915
+
916
+ ```yml
917
+ defaults:
918
+ seed_descriptions:
919
+ enum:
920
+ description: "{{ source_name }} {{ translated_table_name }} {{ translated_attribute_name }} enum"
921
+
922
+ ```
923
+
924
+ #### Generate dbt Seed Enum Files
925
+
926
+ Generate seed enum files for dbt:
927
+
928
+ ```bash
929
+ $ bin/rails generate active_record:dbt:enum TABLE_NAME ENUM_COLUMN_NAME
930
+ ```
931
+
932
+ Generate seed enum files for dbt from the specified `TABLE_NAME` and `ENUM_COLUMN_NAME`.
933
+
934
+ File | Description
935
+ --------- | ---------
936
+ `#{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.csv` | Seed enum file for dbt.
937
+ `#{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.yml` | Seed enum documentation file for dbt.
938
+
939
+ Example:
940
+
941
+ ```bash
942
+ $ bin/rails generate active_record:dbt:enum posts status
943
+ ```
944
+
945
+ ##### Generate `#{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.csv`
946
+
947
+ Example:
948
+
949
+ ```csv
950
+ status_before_type_of_cast,status_key,status_en,status_ja
951
+ 0,draft,Draft,下書き
952
+ 1,published,Published,公開
953
+ 2,deleted,Deleted,削除
954
+
955
+ ```
956
+
957
+ ##### Generate `#{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.yml`
958
+
959
+ Example:
960
+
961
+ > [!NOTE]
962
+ >
963
+ > The output will be as shown below. It is recommended to indent the YAML file with a tool of your choice.
964
+
965
+ ```yaml
966
+ ---
967
+ version: 2
968
+ seeds:
969
+ - name: seed_dummy__post_enum_statuses
970
+ description: dummy Post Status enum
971
+ config:
972
+ column_types:
973
+ status_before_type_of_cast: int64
974
+ status_key: string
975
+ status_en: string
976
+ status_ja: string
977
+ columns:
978
+ - name: status_before_type_of_cast
979
+ description: Status
980
+ tests:
981
+ - unique
982
+ - not_null
983
+ - name: status_key
984
+ description: Status(key)
985
+ tests:
986
+ - unique
987
+ - not_null
988
+ - name: status_en
989
+ description: Status(en)
990
+ tests:
991
+ - unique
992
+ - not_null
993
+ - name: status_ja
994
+ description: Status(ja)
995
+ tests:
996
+ - unique
997
+ - not_null
998
+
999
+ ```
1000
+
886
1001
  ## Contributing
887
1002
 
888
1003
  Contribution directions go here.
@@ -5,6 +5,7 @@ module ActiveRecord
5
5
  module Column
6
6
  class Column
7
7
  include ActiveRecord::Dbt::DataType::Mapper
8
+ include ActiveRecord::Dbt::I18nWrapper::Translate
8
9
 
9
10
  attr_reader :table_name, :column, :column_test, :primary_keys
10
11
 
@@ -23,7 +24,7 @@ module ActiveRecord
23
24
  {
24
25
  'name' => column_name,
25
26
  'description' => description,
26
- 'data_type' => data_type,
27
+ 'data_type' => data_type(column.type),
27
28
  **column_overrides.except(:tests),
28
29
  'tests' => column_test.config
29
30
  }.compact
@@ -45,18 +46,6 @@ module ActiveRecord
45
46
  source_config.dig(:table_descriptions, table_name, :columns, column_name)
46
47
  end
47
48
 
48
- def translated_attribute_name
49
- translated_column_name || translated_default_column_name
50
- end
51
-
52
- def translated_column_name
53
- I18n.t("activerecord.attributes.#{table_name.singularize}.#{column_name}", default: nil)
54
- end
55
-
56
- def translated_default_column_name
57
- I18n.t("attributes.#{column_name}", default: nil)
58
- end
59
-
60
49
  def key_column_name
61
50
  column_name if primary_key? || foreign_key?
62
51
  end
@@ -10,7 +10,13 @@ module ActiveRecord
10
10
  define_required_methods :column
11
11
 
12
12
  def not_null_test
13
- column.null == true ? nil : 'not_null'
13
+ null? ? nil : 'not_null'
14
+ end
15
+
16
+ private
17
+
18
+ def null?
19
+ column.null == true
14
20
  end
15
21
  end
16
22
  end
@@ -9,38 +9,10 @@ module ActiveRecord
9
9
 
10
10
  include ActiveRecord::Dbt::Configuration::DataSync
11
11
  include ActiveRecord::Dbt::Configuration::DwhPlatform
12
+ include ActiveRecord::Dbt::Configuration::I18nConfiguration
12
13
  include ActiveRecord::Dbt::Configuration::Logger
13
- include ActiveRecord::Dbt::Configuration::Parser
14
+ include ActiveRecord::Dbt::Configuration::Source
14
15
  include ActiveRecord::Dbt::Configuration::UsedDbtPackage
15
-
16
- DEFAULT_CONFIG_DIRECTORY_PATH = 'lib/dbt'
17
- DEFAULT_EXPORT_DIRECTORY_PATH = 'doc/dbt'
18
-
19
- attr_writer :config_directory_path, :export_directory_path
20
-
21
- def source_config_path
22
- @source_config_path ||= "#{config_directory_path}/source_config.yml"
23
- end
24
-
25
- def source_config
26
- @source_config ||= parse_yaml(source_config_path)
27
- end
28
-
29
- def source_name
30
- @source_name ||= source_config.dig(:sources, :name).tap do |source_name|
31
- raise SourceNameIsNullError, "'sources.name' is required in '#{source_config_path}'." if source_name.nil?
32
- end
33
- end
34
-
35
- def config_directory_path
36
- @config_directory_path ||= DEFAULT_CONFIG_DIRECTORY_PATH
37
- end
38
-
39
- def export_directory_path
40
- @export_directory_path ||= DEFAULT_EXPORT_DIRECTORY_PATH
41
- end
42
-
43
- class SourceNameIsNullError < StandardError; end
44
16
  end
45
17
  end
46
18
  end
@@ -0,0 +1,14 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Configuration
6
+ module I18nConfiguration
7
+ def locale=(locale = I18n.locale)
8
+ I18n.load_path += Dir[Rails.root.join('config/locales/**/*.{rb,yml}')]
9
+ I18n.locale = locale
10
+ end
11
+ end
12
+ end
13
+ end
14
+ end
@@ -0,0 +1,40 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Configuration
6
+ module Source
7
+ include ActiveRecord::Dbt::Configuration::Parser
8
+
9
+ DEFAULT_CONFIG_DIRECTORY_PATH = 'lib/dbt'
10
+ DEFAULT_EXPORT_DIRECTORY_PATH = 'doc/dbt'
11
+
12
+ attr_writer :config_directory_path, :export_directory_path
13
+
14
+ def config_directory_path
15
+ @config_directory_path ||= DEFAULT_CONFIG_DIRECTORY_PATH
16
+ end
17
+
18
+ def export_directory_path
19
+ @export_directory_path ||= DEFAULT_EXPORT_DIRECTORY_PATH
20
+ end
21
+
22
+ def source_config_path
23
+ @source_config_path ||= "#{config_directory_path}/source_config.yml"
24
+ end
25
+
26
+ def source_config
27
+ @source_config ||= parse_yaml(source_config_path)
28
+ end
29
+
30
+ def source_name
31
+ @source_name ||= source_config.dig(:sources, :name).tap do |source_name|
32
+ raise SourceNameIsNullError, "'sources.name' is required in '#{source_config_path}'." if source_name.nil?
33
+ end
34
+ end
35
+
36
+ class SourceNameIsNullError < StandardError; end
37
+ end
38
+ end
39
+ end
40
+ end
@@ -4,31 +4,98 @@ module ActiveRecord
4
4
  module Dbt
5
5
  module DataType
6
6
  module Mapper
7
- include ActiveRecord::Dbt::DataType::DwhPlatform::ApacheSpark
8
- include ActiveRecord::Dbt::DataType::DwhPlatform::Bigquery
9
- include ActiveRecord::Dbt::DataType::DwhPlatform::PostgreSql
10
- include ActiveRecord::Dbt::DataType::DwhPlatform::Redshift
11
- include ActiveRecord::Dbt::DataType::DwhPlatform::Snowflake
12
-
13
7
  extend ActiveRecord::Dbt::RequiredMethods
14
8
 
15
9
  # [Platform-specific data types | dbt Developer Hub](https://docs.getdbt.com/reference/resource-properties/data-types)
16
10
  RUBY_TO_DWH_PLATFORM_TYPE_MAP = {
17
- 'bigquery' => RUBY_TO_BIGQUERY_TYPES,
18
- 'postgres' => RUBY_TO_POSTGRES_TYPES,
19
- 'redshift' => RUBY_TO_REDSHIFT_TYPES,
20
- 'snowflake' => RUBY_TO_SNOWFLAKE_TYPES,
21
- 'spark' => RUBY_TO_SPARK_TYPES
11
+ # [Data types  |  BigQuery  |  Google Cloud](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#data_type_list)
12
+ 'bigquery' => {
13
+ binary: 'bytes',
14
+ boolean: 'bool',
15
+ date: 'date',
16
+ datetime: 'datetime',
17
+ decimal: 'int64',
18
+ float: 'float64',
19
+ integer: 'int64',
20
+ json: 'json',
21
+ string: 'string',
22
+ text: 'string',
23
+ time: 'time'
24
+ },
25
+
26
+ # TODO: I have not tried it. I don't know if this is the correct data_type.
27
+ # [PostgreSQL: Documentation: 16: Chapter 8. Data Types](https://www.postgresql.org/docs/current/datatype.html)
28
+ 'postgres' => {
29
+ binary: 'bytea',
30
+ boolean: 'boolean',
31
+ date: 'date',
32
+ datetime: 'timestamp',
33
+ decimal: 'bigint',
34
+ float: 'double precision',
35
+ integer: 'bigint',
36
+ json: 'json',
37
+ string: 'text',
38
+ text: 'text',
39
+ time: 'time'
40
+ },
41
+
42
+ # TODO: I have not tried it. I don't know if this is the correct data_type.
43
+ # [Data types - Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/c_Supported_data_types.html)
44
+ 'redshift' => {
45
+ binary: 'varbyte',
46
+ boolean: 'bool',
47
+ date: 'date',
48
+ datetime: 'timestamp',
49
+ decimal: 'decimal',
50
+ float: 'double precision',
51
+ integer: 'integer',
52
+ json: 'super',
53
+ string: 'varchar',
54
+ text: 'varchar',
55
+ time: 'time'
56
+ },
57
+
58
+ # TODO: I have not tried it. I don't know if this is the correct data_type.
59
+ # [Summary of data types | Snowflake Documentation](https://docs.snowflake.com/en/sql-reference/intro-summary-data-types)
60
+ 'snowflake' => {
61
+ binary: 'binary',
62
+ boolean: 'boolean',
63
+ date: 'date',
64
+ datetime: 'datetime',
65
+ decimal: 'decimal',
66
+ float: 'float',
67
+ integer: 'integer',
68
+ json: 'variant',
69
+ string: 'varchar',
70
+ text: 'varchar',
71
+ time: 'time'
72
+ },
73
+
74
+ # TODO: I have not tried it. I don't know if this is the correct data_type.
75
+ # [Data Types - Spark 3.5.1 Documentation](https://spark.apache.org/docs/latest/sql-ref-datatypes.html)
76
+ 'spark' => {
77
+ binary: 'binary',
78
+ boolean: 'boolean',
79
+ date: 'date',
80
+ datetime: 'timestamp',
81
+ decimal: 'decimal',
82
+ float: 'float',
83
+ integer: 'integer',
84
+ json: 'string',
85
+ string: 'string',
86
+ text: 'string',
87
+ time: 'string'
88
+ }
22
89
  }.freeze
23
90
 
24
- define_required_methods :column, :@config
91
+ define_required_methods :@config
25
92
 
26
93
  delegate :dwh_platform, to: :@config
27
94
 
28
95
  private
29
96
 
30
- def data_type
31
- RUBY_TO_DWH_PLATFORM_TYPE_MAP[dwh_platform].fetch(column.type, 'unknown')
97
+ def data_type(type)
98
+ RUBY_TO_DWH_PLATFORM_TYPE_MAP[dwh_platform].fetch(type, 'unknown')
32
99
  end
33
100
  end
34
101
  end
@@ -14,13 +14,12 @@ module ActiveRecord
14
14
  delegate :used_dbterd?, :add_log, to: :@config
15
15
 
16
16
  def relationships_meta_relationship_type
17
- return nil unless used_dbterd?
18
- return nil if no_relationship?
17
+ return nil if !used_dbterd? || no_relationship?
19
18
 
20
19
  {
21
20
  'relationship_type' => relationship_type
22
21
  }
23
- rescue NotSpecifiedOrNotInvalidIdError, StandardError => e
22
+ rescue NotSpecifiedOrNotInvalidIdError, NameError => e
24
23
  relationships_meta_relationship_type_with_active_record_dbt_error(e)
25
24
  end
26
25
 
@@ -5,14 +5,14 @@ module ActiveRecord
5
5
  module Factory
6
6
  module Model
7
7
  module StagingFactory
8
- def self.yml_build(table_name)
8
+ def self.build(table_name)
9
9
  table_factory = ActiveRecord::Dbt::Factory::TableFactory.build(table_name)
10
10
  yml = ActiveRecord::Dbt::Model::Staging::Yml.new(table_factory)
11
- struct = Struct.new(:export_path, :yml_dump, keyword_init: true)
11
+ struct = Struct.new(:export_path, :dump, keyword_init: true)
12
12
 
13
13
  struct.new(
14
14
  export_path: yml.export_path,
15
- yml_dump: YAML.dump(yml.config.deep_stringify_keys)
15
+ dump: yml.dump
16
16
  )
17
17
  end
18
18
  end
@@ -6,9 +6,8 @@ module ActiveRecord
6
6
  module SourceFactory
7
7
  def self.build
8
8
  tables_factory = ActiveRecord::Dbt::Factory::TablesFactory.build
9
- config = ActiveRecord::Dbt::Source::Yml.new(tables_factory).config
10
9
 
11
- YAML.dump(config.deep_stringify_keys)
10
+ ActiveRecord::Dbt::Source::Yml.new(tables_factory).dump
12
11
  end
13
12
  end
14
13
  end
@@ -0,0 +1,34 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module I18nWrapper
6
+ module Translate
7
+ extend ActiveRecord::Dbt::RequiredMethods
8
+
9
+ define_required_methods :table_name
10
+
11
+ delegate :singularize, to: :table_name, prefix: true
12
+
13
+ def translated_table_name
14
+ I18n.t("activerecord.models.#{table_name_singularize}", default: nil)
15
+ end
16
+
17
+ def translated_attribute_name
18
+ @translated_attribute_name ||=
19
+ translated_column_name || translated_default_column_name
20
+ end
21
+
22
+ private
23
+
24
+ def translated_column_name
25
+ I18n.t("activerecord.attributes.#{table_name_singularize}.#{column_name}", default: nil)
26
+ end
27
+
28
+ def translated_default_column_name
29
+ I18n.t("attributes.#{column_name}", default: nil)
30
+ end
31
+ end
32
+ end
33
+ end
34
+ end
@@ -21,7 +21,13 @@ module ActiveRecord
21
21
  "#{basename}.yml"
22
22
  end
23
23
 
24
- def config
24
+ def dump
25
+ YAML.dump(model_config.deep_stringify_keys)
26
+ end
27
+
28
+ private
29
+
30
+ def model_config
25
31
  {
26
32
  'version' => 2,
27
33
  'models' => [
@@ -34,8 +40,6 @@ module ActiveRecord
34
40
  }
35
41
  end
36
42
 
37
- private
38
-
39
43
  def columns
40
44
  @columns ||= sort_columns(table.config['columns'])
41
45
  end
@@ -0,0 +1,48 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Seed
6
+ module Enum
7
+ module Base
8
+ attr_reader :table_name, :enum_column_name
9
+
10
+ delegate :source_name, :export_directory_path, to: :@config
11
+ delegate :singularize, to: :table_name, prefix: true
12
+
13
+ def initialize(table_name, enum_column_name)
14
+ @table_name = table_name
15
+ @enum_column_name = enum_column_name
16
+ @config = ActiveRecord::Dbt::Config.instance
17
+ end
18
+
19
+ private
20
+
21
+ def basename
22
+ "#{export_directory_path}/#{seed_name}"
23
+ end
24
+
25
+ def seed_name
26
+ "seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}"
27
+ end
28
+
29
+ def locales
30
+ @locales ||= I18n.available_locales
31
+ end
32
+
33
+ def application_record_klass
34
+ @application_record_klass ||= table_name_singularize.classify.constantize
35
+ rescue NameError => _e
36
+ raise DoesNotExistTableError, "#{table_name} table does not exist."
37
+ end
38
+
39
+ def enum_pluralized
40
+ enum_column_name.pluralize
41
+ end
42
+
43
+ class DoesNotExistTableError < StandardError; end
44
+ end
45
+ end
46
+ end
47
+ end
48
+ end
@@ -0,0 +1,85 @@
1
+ # frozen_string_literal: true
2
+
3
+ require 'csv'
4
+
5
+ module ActiveRecord
6
+ module Dbt
7
+ module Seed
8
+ module Enum
9
+ class Csv
10
+ include ActiveRecord::Dbt::Seed::Enum::Base
11
+
12
+ delegate :singularize, to: :table_name, prefix: true
13
+
14
+ def export_path
15
+ "#{basename}.csv"
16
+ end
17
+
18
+ def dump
19
+ CSV.generate(headers: true) do |csv|
20
+ rows.each { |row| csv << row }
21
+ end
22
+ end
23
+
24
+ private
25
+
26
+ def rows
27
+ [
28
+ header,
29
+ *enum_rows
30
+ ]
31
+ end
32
+
33
+ def header
34
+ [
35
+ "#{enum_column_name}_before_type_of_cast",
36
+ "#{enum_column_name}_key",
37
+ *locale_header
38
+ ]
39
+ end
40
+
41
+ def locale_header
42
+ locales.map { |locale| "#{enum_column_name}_#{locale}" }
43
+ end
44
+
45
+ def enum_rows
46
+ enums.map do |enum_key, enum_before_type_of_cast|
47
+ [
48
+ enum_before_type_of_cast,
49
+ enum_key,
50
+ *enum_values(enum_key)
51
+ ]
52
+ end
53
+ end
54
+
55
+ def enums
56
+ application_record_klass.defined_enums.fetch(enum_column_name)
57
+ rescue KeyError => _e
58
+ raise NotEnumColumnError, "#{table_name}.#{enum_column_name} column is not an enum." if enum_column?
59
+
60
+ raise DoesNotExistColumnError, "The #{enum_column_name} column in the #{table_name} table does not exist."
61
+ end
62
+
63
+ def enum_column?
64
+ application_record_klass.column_names.include?(enum_column_name)
65
+ end
66
+
67
+ def enum_values(enum_key)
68
+ locales.map { |locale| translated_enum_value(enum_key, locale) }
69
+ end
70
+
71
+ def translated_enum_value(enum_key, locale)
72
+ I18n.t(
73
+ "activerecord.enum.#{table_name_singularize}.#{enum_column_name}.#{enum_key}",
74
+ locale: locale,
75
+ default: nil
76
+ )
77
+ end
78
+
79
+ class DoesNotExistColumnError < StandardError; end
80
+ class NotEnumColumnError < StandardError; end
81
+ end
82
+ end
83
+ end
84
+ end
85
+ end
@@ -0,0 +1,128 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Seed
6
+ module Enum
7
+ class Yml
8
+ include ActiveRecord::Dbt::Column::Testable::UniqueTestable
9
+ include ActiveRecord::Dbt::Column::Testable::NotNullTestable
10
+ include ActiveRecord::Dbt::DataType::Mapper
11
+ include ActiveRecord::Dbt::I18nWrapper::Translate
12
+ include ActiveRecord::Dbt::Seed::Enum::Base
13
+
14
+ delegate :source_config, to: :@config
15
+
16
+ alias column_name enum_column_name
17
+
18
+ def export_path
19
+ "#{basename}.yml"
20
+ end
21
+
22
+ def dump
23
+ YAML.dump(seed_config.deep_stringify_keys)
24
+ end
25
+
26
+ private
27
+
28
+ def seed_config
29
+ {
30
+ 'version' => 2,
31
+ 'seeds' => [
32
+ 'name' => seed_name,
33
+ 'description' => seed_description,
34
+ 'config' => {
35
+ 'column_types' => column_types
36
+ }
37
+ ],
38
+ 'columns' => columns
39
+ }
40
+ end
41
+
42
+ def seed_description
43
+ default_seed_description ||
44
+ "#{source_name} #{translated_table_name} #{translated_attribute_name} enum".strip
45
+ end
46
+
47
+ def default_seed_description
48
+ source_config.dig(:defaults, :seed_descriptions, :enum, :description)
49
+ &.gsub(/{{\s*source_name\s*}}/, source_name)
50
+ &.gsub(/{{\s*translated_table_name\s*}}/, translated_table_name)
51
+ &.gsub(/{{\s*translated_attribute_name\s*}}/, translated_attribute_name)
52
+ end
53
+
54
+ def column_types
55
+ {
56
+ "#{enum_column_name}_before_type_of_cast" => data_type(before_type_of_cast_type),
57
+ "#{enum_column_name}_key" => data_type(:string),
58
+ **enum_column_types
59
+ }
60
+ end
61
+
62
+ def before_type_of_cast_type
63
+ application_record_klass.columns_hash[enum_column_name].type
64
+ end
65
+
66
+ def enum_column_types
67
+ locales.each_with_object({}) do |locale, hash|
68
+ hash["#{enum_column_name}_#{locale}"] = data_type(:string)
69
+ end
70
+ end
71
+
72
+ def columns
73
+ [
74
+ before_type_of_cast_column,
75
+ enum_key_column,
76
+ *enum_columns
77
+ ].compact
78
+ end
79
+
80
+ def before_type_of_cast_column
81
+ {
82
+ 'name' => "#{enum_column_name}_before_type_of_cast",
83
+ 'description' => translated_attribute_name,
84
+ 'tests' => tests
85
+ }.compact
86
+ end
87
+
88
+ def enum_key_column
89
+ {
90
+ 'name' => "#{enum_column_name}_key",
91
+ 'description' => "#{translated_attribute_name}(key)",
92
+ 'tests' => tests
93
+ }.compact
94
+ end
95
+
96
+ def enum_columns
97
+ locales.each_with_object([]) do |locale, array|
98
+ array.push(
99
+ {
100
+ 'name' => "#{enum_column_name}_#{locale}",
101
+ 'description' => "#{translated_attribute_name}(#{locale})",
102
+ 'tests' => tests
103
+ }.compact
104
+ )
105
+ end
106
+ end
107
+
108
+ def tests
109
+ [
110
+ unique_test,
111
+ not_null_test
112
+ ].compact.presence
113
+ end
114
+
115
+ # MEMO: I think all enums are unique.
116
+ def unique?
117
+ true
118
+ end
119
+
120
+ # MEMO: I think all enums are null.
121
+ def null?
122
+ false
123
+ end
124
+ end
125
+ end
126
+ end
127
+ end
128
+ end
@@ -13,6 +13,12 @@ module ActiveRecord
13
13
  @config = ActiveRecord::Dbt::Config.instance
14
14
  end
15
15
 
16
+ def dump
17
+ YAML.dump(config.deep_stringify_keys)
18
+ end
19
+
20
+ private
21
+
16
22
  def config
17
23
  {
18
24
  'version' => 2,
@@ -22,8 +28,6 @@ module ActiveRecord
22
28
  }
23
29
  end
24
30
 
25
- private
26
-
27
31
  def source_properties
28
32
  source_config[:sources]
29
33
  end
@@ -4,6 +4,7 @@ module ActiveRecord
4
4
  module Dbt
5
5
  module Table
6
6
  class Yml
7
+ include ActiveRecord::Dbt::I18nWrapper::Translate
7
8
  include ActiveRecord::Dbt::Table::Base
8
9
 
9
10
  attr_reader :table_test, :columns
@@ -51,10 +52,6 @@ module ActiveRecord
51
52
  "Write a logical_name of the '#{table_name}' table."
52
53
  end
53
54
 
54
- def translated_table_name
55
- I18n.t("activerecord.models.#{table_name.singularize}", default: nil)
56
- end
57
-
58
55
  def default_logical_name
59
56
  source_config.dig(:defaults, :table_descriptions, :logical_name)
60
57
  &.gsub(/{{\s*table_name\s*}}/, table_name)
@@ -2,6 +2,6 @@
2
2
 
3
3
  module ActiveRecord
4
4
  module Dbt
5
- VERSION = '0.2.0'
5
+ VERSION = '0.3.0'
6
6
  end
7
7
  end
@@ -40,6 +40,9 @@ defaults:
40
40
  logical_name: Write a logical_name of the '{{ table_name }}' table.
41
41
  columns:
42
42
  description: Write a description of the '{{ table_name }}.{{ column_name }}' column.
43
+ seed_descriptions:
44
+ enum:
45
+ description: "{{ source_name }} {{ translated_table_name }} {{ translated_attribute_name }} enum"
43
46
 
44
47
  table_descriptions:
45
48
  ar_internal_metadata:
@@ -0,0 +1,9 @@
1
+ Description:
2
+ Generate seed enum files for dbt from the specified TABLE_NAME and ENUM_COLUMN_NAME.
3
+
4
+ Example:
5
+ bin/rails generate active_record:dbt:enum TABLE_NAME ENUM_COLUMN_NAME
6
+
7
+ This will create:
8
+ #{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.csv
9
+ #{export_directory_path}/seed_#{source_name}__#{table_name_singularize}_enum_#{enum_pluralized}.yml
@@ -0,0 +1,31 @@
1
+ # frozen_string_literal: true
2
+
3
+ module ActiveRecord
4
+ module Dbt
5
+ module Generators
6
+ class EnumGenerator < Rails::Generators::NamedBase
7
+ source_root File.expand_path('templates', __dir__)
8
+
9
+ argument :enum_column_name, type: :string, default: nil
10
+
11
+ def create_enum_seed_csv_file
12
+ create_file csv.export_path, csv.dump
13
+ end
14
+
15
+ def create_enum_seed_yml_file
16
+ create_file yml.export_path, yml.dump
17
+ end
18
+
19
+ private
20
+
21
+ def csv
22
+ @csv ||= ActiveRecord::Dbt::Seed::Enum::Csv.new(name, enum_column_name)
23
+ end
24
+
25
+ def yml
26
+ @yml ||= ActiveRecord::Dbt::Seed::Enum::Yml.new(name, enum_column_name)
27
+ end
28
+ end
29
+ end
30
+ end
31
+ end
@@ -11,7 +11,7 @@ module ActiveRecord
11
11
  end
12
12
 
13
13
  def create_staging_model_yml_file
14
- create_file yml.export_path, yml.yml_dump
14
+ create_file yml.export_path, yml.dump
15
15
  end
16
16
 
17
17
  private
@@ -21,7 +21,7 @@ module ActiveRecord
21
21
  end
22
22
 
23
23
  def yml
24
- @yml ||= ActiveRecord::Dbt::Factory::Model::StagingFactory.yml_build(name)
24
+ @yml ||= ActiveRecord::Dbt::Factory::Model::StagingFactory.build(name)
25
25
  end
26
26
 
27
27
  def source_paths
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: activerecord-dbt
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - yamotech
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2024-08-03 00:00:00.000000000 Z
11
+ date: 2024-08-14 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activerecord
@@ -73,14 +73,11 @@ files:
73
73
  - lib/active_record/dbt/config.rb
74
74
  - lib/active_record/dbt/configuration/data_sync.rb
75
75
  - lib/active_record/dbt/configuration/dwh_platform.rb
76
+ - lib/active_record/dbt/configuration/i18n_configuration.rb
76
77
  - lib/active_record/dbt/configuration/logger.rb
77
78
  - lib/active_record/dbt/configuration/parser.rb
79
+ - lib/active_record/dbt/configuration/source.rb
78
80
  - lib/active_record/dbt/configuration/used_dbt_package.rb
79
- - lib/active_record/dbt/data_type/dwh_platform/apache_spark.rb
80
- - lib/active_record/dbt/data_type/dwh_platform/bigquery.rb
81
- - lib/active_record/dbt/data_type/dwh_platform/postgre_sql.rb
82
- - lib/active_record/dbt/data_type/dwh_platform/redshift.rb
83
- - lib/active_record/dbt/data_type/dwh_platform/snowflake.rb
84
81
  - lib/active_record/dbt/data_type/mapper.rb
85
82
  - lib/active_record/dbt/dbt_package/dbt_utils/table/testable/unique_combination_of_columns_testable.rb
86
83
  - lib/active_record/dbt/dbt_package/dbterd/column/testable/relationships_meta_relationship_type.rb
@@ -89,11 +86,15 @@ files:
89
86
  - lib/active_record/dbt/factory/source_factory.rb
90
87
  - lib/active_record/dbt/factory/table_factory.rb
91
88
  - lib/active_record/dbt/factory/tables_factory.rb
89
+ - lib/active_record/dbt/i18n_wrapper/translate.rb
92
90
  - lib/active_record/dbt/model/staging/base.rb
93
91
  - lib/active_record/dbt/model/staging/sql.rb
94
92
  - lib/active_record/dbt/model/staging/yml.rb
95
93
  - lib/active_record/dbt/railtie.rb
96
94
  - lib/active_record/dbt/required_methods.rb
95
+ - lib/active_record/dbt/seed/enum/base.rb
96
+ - lib/active_record/dbt/seed/enum/csv.rb
97
+ - lib/active_record/dbt/seed/enum/yml.rb
97
98
  - lib/active_record/dbt/source/yml.rb
98
99
  - lib/active_record/dbt/table/base.rb
99
100
  - lib/active_record/dbt/table/test.rb
@@ -102,6 +103,8 @@ files:
102
103
  - lib/generators/active_record/dbt/config/USAGE
103
104
  - lib/generators/active_record/dbt/config/config_generator.rb
104
105
  - lib/generators/active_record/dbt/config/templates/source_config.yml.tt
106
+ - lib/generators/active_record/dbt/enum/USAGE
107
+ - lib/generators/active_record/dbt/enum/enum_generator.rb
105
108
  - lib/generators/active_record/dbt/initializer/USAGE
106
109
  - lib/generators/active_record/dbt/initializer/initializer_generator.rb
107
110
  - lib/generators/active_record/dbt/initializer/templates/dbt.rb
@@ -1,27 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module ApacheSpark
8
- # TODO: I have not tried it. I don't know if this is the correct data_type.
9
- # [Data Types - Spark 3.5.1 Documentation](https://spark.apache.org/docs/latest/sql-ref-datatypes.html)
10
- RUBY_TO_SPARK_TYPES = {
11
- binary: 'binary',
12
- boolean: 'boolean',
13
- date: 'date',
14
- datetime: 'timestamp',
15
- decimal: 'decimal',
16
- float: 'float',
17
- integer: 'integer',
18
- json: 'string',
19
- string: 'string',
20
- text: 'string',
21
- time: 'string'
22
- }.freeze
23
- end
24
- end
25
- end
26
- end
27
- end
@@ -1,26 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module Bigquery
8
- # [Data types  |  BigQuery  |  Google Cloud](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#data_type_list)
9
- RUBY_TO_BIGQUERY_TYPES = {
10
- binary: 'bytes',
11
- boolean: 'bool',
12
- date: 'date',
13
- datetime: 'datetime',
14
- decimal: 'int64',
15
- float: 'float64',
16
- integer: 'int64',
17
- json: 'json',
18
- string: 'string',
19
- text: 'string',
20
- time: 'time'
21
- }.freeze
22
- end
23
- end
24
- end
25
- end
26
- end
@@ -1,27 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module PostgreSql
8
- # TODO: I have not tried it. I don't know if this is the correct data_type.
9
- # [PostgreSQL: Documentation: 16: Chapter 8. Data Types](https://www.postgresql.org/docs/current/datatype.html)
10
- RUBY_TO_POSTGRES_TYPES = {
11
- binary: 'bytea',
12
- boolean: 'boolean',
13
- date: 'date',
14
- datetime: 'timestamp',
15
- decimal: 'bigint',
16
- float: 'double precision',
17
- integer: 'bigint',
18
- json: 'json',
19
- string: 'text',
20
- text: 'text',
21
- time: 'time'
22
- }.freeze
23
- end
24
- end
25
- end
26
- end
27
- end
@@ -1,27 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module Redshift
8
- # TODO: I have not tried it. I don't know if this is the correct data_type.
9
- # [Data types - Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/c_Supported_data_types.html)
10
- RUBY_TO_REDSHIFT_TYPES = {
11
- binary: 'varbyte',
12
- boolean: 'bool',
13
- date: 'date',
14
- datetime: 'timestamp',
15
- decimal: 'decimal',
16
- float: 'double precision',
17
- integer: 'integer',
18
- json: 'super',
19
- string: 'varchar',
20
- text: 'varchar',
21
- time: 'time'
22
- }.freeze
23
- end
24
- end
25
- end
26
- end
27
- end
@@ -1,27 +0,0 @@
1
- # frozen_string_literal: true
2
-
3
- module ActiveRecord
4
- module Dbt
5
- module DataType
6
- module DwhPlatform
7
- module Snowflake
8
- # TODO: I have not tried it. I don't know if this is the correct data_type.
9
- # [Summary of data types | Snowflake Documentation](https://docs.snowflake.com/en/sql-reference/intro-summary-data-types)
10
- RUBY_TO_SNOWFLAKE_TYPES = {
11
- binary: 'binary',
12
- boolean: 'boolean',
13
- date: 'date',
14
- datetime: 'datetime',
15
- decimal: 'decimal',
16
- float: 'float',
17
- integer: 'integer',
18
- json: 'variant',
19
- string: 'varchar',
20
- text: 'varchar',
21
- time: 'time'
22
- }.freeze
23
- end
24
- end
25
- end
26
- end
27
- end