activerecord-dbt 0.4.0 → 0.5.0

data/README.md

@@ -6,7 +6,33 @@
 
 `ActiveRecord::Dbt` generates [dbt](https://www.getdbt.com/) files from the information of the database connected via ActiveRecord.
 
-Currently, it can generate `yaml` files for `sources` and `models` files for `staging`.
+Currently, it can generate:
+
+- Sources: `YAML` files
+- Staging models: `SQL` and `YAML` files
+- Seeds: `CSV` and `YAML` files (based on `ActiveRecord` enum values)
+
+> [!WARNING]
+>
+> dbt Version Compatibility
+>
+> Generated files are optimized for dbt v1.10+. For older versions, the following manual changes are required:
+>
+> | dbt Version | Required Changes |
+> |-------------|------------------|
+> | v1.7 or below | Rename `data_tests:` to `tests:` ([reference](https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax)) |
+> | v1.9 or below | Move properties from under `config:` to the top level ([reference](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.10#custom-inputs)) |
+
+## Table of Contents
+
+- [Installation](#installation)
+- [Usage](#usage)
+  - [Configuration](#configuration)
+  - [Generate dbt Source File](#generate-dbt-source-file)
+  - [Generate dbt Staging Files](#generate-dbt-staging-files)
+  - [Generate dbt Seed Files](#generate-dbt-seed-files)
+- [Contributing](#contributing)
+- [License](#license)
 
 ## Installation
 
@@ -52,8 +78,8 @@ This will generate the `config/initializers/dbt.rb` file.
 
 Configuration | Description
 --------- | ---------
-config_directory_path | The path to the directory where files generated by `bin/rails generate active_record:dbt:*` are stored. The default is `lib/dbt`.
-export_directory_path | The path to the directory where configuration files are stored. The default is `doc/dbt`.
+config_directory_path | The path to the directory where configuration files are stored. The default is `lib/dbt`.
+export_directory_path | The path to the directory where files generated by `bin/rails generate active_record:dbt:*` are stored. The default is `doc/dbt`.
 dwh_platform | Specify the data warehouse platform to which dbt connects. The default is `bigquery`.
 data_sync_delayed | Indicates whether there is a data delay. If set to `true`, `severity: warn` is applied to the `relationships` test. The default is `false`.
 logger | The destination for log output. The default is `Logger.new('./log/active_record_dbt.log')`.
@@ -126,18 +152,29 @@ The available properties for `sources` and `table_overrides` are detailed in [So
 
 ##### sources
 
-Set all properties except for `tables`.
+Set the configurations for `sources`.
+
+Configuration | Description
+--------- | ---------
+project_name | Written to the beginning of the table or model `description`.
+table_names in `exclude` | Specify which table names you do not want output in `sources`.
 
 Example:
 
 ```yml
 sources:
   name: dummy
-  meta:
-    generated_by: activerecord-dbt
   description: |-
     Write a description of the 'dummy' source.
     You can write multiple lines.
+  config:
+    meta:
+      project_name: dummy_project
+      generated_by: activerecord-dbt
+      exclude:
+        table_names:
+          - ar_internal_metadata
+          - schema_migrations
 
 ```
 
@@ -150,14 +187,15 @@ Example:
 ```yml
 table_overrides:
   users:
-    loaded_at_field: created_at
-    freshness:
-      warn_after:
-        count: 3
-        period: day
-      error_after:
-        count: 5
-        period: day
+    config:
+      freshness:
+        warn_after:
+          count: 3
+          period: day
+        error_after:
+          count: 5
+          period: day
+      loaded_at_field: created_at
     columns:
       created_at:
         data_tests:
@@ -170,15 +208,15 @@ table_overrides:
 
 Set the default value for the `description`(`logical_name`, `description`) of `tables`.
 
-In the `logical_name` and `description` of `table_descriptions`, you can refer to the table name with `{{ table_name }}`.
-In the `description` of `table_descriptions.columns`, you can refer to the table name with `{{ table_name }}` and the column name with `{{ column_name }}`.
+In the `logical_name` and `description` fields of `table_descriptions`, you can use `{{ table_name }}` to refer to the table name and `{{ project_name }}` to refer to the project name.
+In the `description` of `table_descriptions.columns`, you can use `{{ table_name }}` to refer to the table name and `{{ column_name }}` to refer to the column name.
 
 Example:
 
 ```yml
 defaults:
   table_descriptions:
-    logical_name: Write a logical_name of the '{{ table_name }}' table.
+    logical_name: Write the logical_name of the '{{ table_name }}' table in '{{ project_name }}'.
     columns:
       description: Write a description of the '{{ table_name }}.{{ column_name }}' column.
 
@@ -189,7 +227,7 @@ If nothing is set, it defaults to the following:
 ```yml
 defaults:
   table_descriptions:
-    logical_name: Write a logical_name of the '{{ table_name }}' table.
+    logical_name: Write the logical_name of the '{{ table_name }}' table in '{{ project_name }}'.
     columns:
       description: Write a description of the '{{ table_name }}.{{ column_name }}' column.
 
@@ -228,29 +266,33 @@ table_descriptions:
 
 ```
 
-##### Example:
-
 Adjust the settings according to your environment.
 
 ```yml
 sources:
   name: dummy
-  meta:
-    generated_by: activerecord-dbt
   description: |-
     Write a description of the 'dummy' source.
     You can write multiple lines.
+  config:
+    meta:
+      project_name: dummy_project
+      generated_by: activerecord-dbt
+      exclude:
+        table_names:
+          - hoges
 
 table_overrides:
   users:
-    loaded_at_field: created_at
-    freshness:
-      warn_after:
-        count: 3
-        period: day
-      error_after:
-        count: 5
-        period: day
+    config:
+      freshness:
+        warn_after:
+          count: 3
+          period: day
+        error_after:
+          count: 5
+          period: day
+      loaded_at_field: created_at
     columns:
       created_at:
         data_tests:
@@ -259,9 +301,12 @@ table_overrides:
 
 defaults:
   table_descriptions:
-    logical_name: Write a logical_name of the '{{ table_name }}' table.
+    logical_name: Write the logical_name of the '{{ table_name }}' table in '{{ project_name }}'.
     columns:
       description: Write a description of the '{{ table_name }}.{{ column_name }}' column.
+  seed_descriptions:
+    enum:
+      description: "{{ project_name }} {{ table_logical_name }} {{ column_description }} enum"
 
 table_descriptions:
   ar_internal_metadata:
@@ -294,34 +339,31 @@ $ bin/rails generate active_record:dbt:source
 
 Generate `#{export_directory_path}/models/sources/#{source_name}/src_#{source_name}.yml`.
 
-##### Example:
+Example:
 
 > [!NOTE]
 >
 > The output will be as shown below. It is recommended to indent the YAML file with a tool of your choice.
 
-> [!WARNING]
->
-> If you are using a version of dbt lower than v1.8, replace `tests:` with `data_tests:` in the generated file.
->
-> [Add data tests to your DAG | dbt Developer Hub](https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax)
->
->> Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, the key was renamed from `tests:` to `data_tests:`.
-
 ```yaml
 ---
 version: 2
 sources:
 - name: dummy
-  meta:
-    generated_by: activerecord-dbt
   description: |-
     Write a description of the 'dummy' source.
     You can write multiple lines.
+  config:
+    meta:
+      project_name: dummy_project
+      generated_by: activerecord-dbt
+      exclude:
+        table_names:
+        - hoges
   tables:
   - name: ar_internal_metadata
     description: |-
-      # Internal Metadata
+      # dummy_project Internal Metadata
       By default Rails will store information about your Rails environment and schema
       in an internal table named `ar_internal_metadata`.
     columns:
@@ -345,7 +387,7 @@ sources:
       data_tests:
       - not_null
   - name: companies
-    description: Write a logical_name of the 'companies' table.
+    description: Write the logical_name of the 'companies' table in 'dummy_project'.
     columns:
     - name: id
       description: id
@@ -370,10 +412,11 @@ sources:
       data_tests:
       - not_null
       - accepted_values:
-          values:
-          - true
-          - false
-          quote: false
+          arguments:
+            values:
+            - true
+            - false
+            quote: false
     - name: created_at
       description: Created At
       data_type: datetime
@@ -385,7 +428,7 @@ sources:
       data_tests:
       - not_null
   - name: posts
-    description: Post
+    description: dummy_project Post
     columns:
     - name: id
       description: ID
@@ -399,10 +442,11 @@ sources:
       data_tests:
       - not_null
       - relationships:
-          to: source('dummy', 'users')
-          field: id
-          meta:
-            relationship_type: many-to-one
+          arguments:
+            to: source('dummy', 'users')
+            field: id
+            meta:
+              relationship_type: many-to-one
     - name: title
       description: Title
       data_type: string
@@ -420,22 +464,24 @@ sources:
       data_tests:
       - not_null
     - name: status
-      description: Write a description of the 'posts.status' column.
+      description: Status
       data_type: int64
       data_tests:
       - accepted_values:
-          values:
-          - 0
-          - 1
-          - 2
-          quote: false
+          arguments:
+            values:
+            - 0
+            - 1
+            - 2
+            quote: false
   - name: posts_tags
-    description: Write a logical_name of the 'posts_tags' table.
+    description: Write the logical_name of the 'posts_tags' table in 'dummy_project'.
     data_tests:
     - dbt_utils.unique_combination_of_columns:
-        combination_of_columns:
-        - post_id
-        - tag_id
+        arguments:
+          combination_of_columns:
+          - post_id
+          - tag_id
     columns:
     - name: post_id
       description: post_id
@@ -443,28 +489,38 @@ sources:
       data_tests:
       - not_null
       - relationships:
-          to: source('dummy', 'posts')
-          field: id
-          meta:
-            relationship_type: many-to-one
-            active_record_dbt_error:
-              class: NameError
-              message: uninitialized constant PostsTag
+          arguments:
+            to: source('dummy', 'posts')
+            field: id
+            meta:
+              relationship_type: many-to-one
+              active_record_dbt_error:
+                class: NameError
+                message: |-
+                  uninitialized constant PostsTag
+
+                        Object.const_get(camel_cased_word)
+                              ^^^^^^^^^^
     - name: tag_id
       description: tag_id
       data_type: int64
       data_tests:
       - not_null
       - relationships:
-          to: source('dummy', 'tags')
-          field: id
-          meta:
-            relationship_type: many-to-one
-            active_record_dbt_error:
-              class: NameError
-              message: uninitialized constant PostsTag
+          arguments:
+            to: source('dummy', 'tags')
+            field: id
+            meta:
+              relationship_type: many-to-one
+              active_record_dbt_error:
+                class: NameError
+                message: |-
+                  uninitialized constant PostsTag
+
+                        Object.const_get(camel_cased_word)
+                              ^^^^^^^^^^
   - name: profiles
-    description: Write a logical_name of the 'profiles' table.
+    description: Write the logical_name of the 'profiles' table in 'dummy_project'.
     columns:
     - name: id
       description: id
@@ -479,10 +535,11 @@ sources:
       - unique
       - not_null
       - relationships:
-          to: source('dummy', 'users')
-          field: id
-          meta:
-            relationship_type: one-to-one
+          arguments:
+            to: source('dummy', 'users')
+            field: id
+            meta:
+              relationship_type: one-to-one
     - name: first_name
       description: Write a description of the 'profiles.first_name' column.
       data_type: string
@@ -504,12 +561,13 @@ sources:
       data_tests:
       - not_null
   - name: relationships
-    description: Write a logical_name of the 'relationships' table.
+    description: Write the logical_name of the 'relationships' table in 'dummy_project'.
     data_tests:
     - dbt_utils.unique_combination_of_columns:
-        combination_of_columns:
-        - follower_id
-        - followed_id
+        arguments:
+          combination_of_columns:
+          - follower_id
+          - followed_id
     columns:
     - name: id
       description: id
@@ -523,20 +581,22 @@ sources:
       data_tests:
       - not_null
       - relationships:
-          to: source('dummy', 'users')
-          field: id
-          meta:
-            relationship_type: many-to-one
+          arguments:
+            to: source('dummy', 'users')
+            field: id
+            meta:
+              relationship_type: many-to-one
     - name: followed_id
       description: followed_id
       data_type: int64
       data_tests:
       - not_null
       - relationships:
-          to: source('dummy', 'users')
-          field: id
-          meta:
-            relationship_type: many-to-one
+          arguments:
+            to: source('dummy', 'users')
+            field: id
+            meta:
+              relationship_type: many-to-one
     - name: created_at
       description: Created At
       data_type: datetime
@@ -549,7 +609,7 @@ sources:
       - not_null
   - name: schema_migrations
     description: |-
-      # Schema Migrations
+      # dummy_project Schema Migrations
       Rails keeps track of which migrations have been committed to the database and
       stores them in a neighboring table in that same database called `schema_migrations`.
     columns:
@@ -560,7 +620,7 @@ sources:
       - unique
       - not_null
   - name: tags
-    description: Write a logical_name of the 'tags' table.
+    description: Write the logical_name of the 'tags' table in 'dummy_project'.
     columns:
     - name: id
       description: id
@@ -585,12 +645,13 @@ sources:
       data_tests:
       - not_null
   - name: user_tags
-    description: Write a logical_name of the 'user_tags' table.
+    description: Write the logical_name of the 'user_tags' table in 'dummy_project'.
     data_tests:
     - dbt_utils.unique_combination_of_columns:
-        combination_of_columns:
-        - user_id
-        - tag_id
+        arguments:
+          combination_of_columns:
+          - user_id
+          - tag_id
     columns:
     - name: id
       description: id
@@ -604,20 +665,22 @@ sources:
       data_tests:
       - not_null
       - relationships:
-          to: source('dummy', 'users')
-          field: id
-          meta:
-            relationship_type: many-to-one
+          arguments:
+            to: source('dummy', 'users')
+            field: id
+            meta:
+              relationship_type: many-to-one
     - name: tag_id
       description: tag_id
       data_type: int64
       data_tests:
       - not_null
       - relationships:
-          to: source('dummy', 'tags')
-          field: id
-          meta:
-            relationship_type: many-to-one
+          arguments:
+            to: source('dummy', 'tags')
+            field: id
+            meta:
+              relationship_type: many-to-one
     - name: created_at
       description: Created At
       data_type: datetime
@@ -629,15 +692,16 @@ sources:
       data_tests:
       - not_null
   - name: users
-    description: User
-    loaded_at_field: created_at
-    freshness:
-      warn_after:
-        count: 3
-        period: day
-      error_after:
-        count: 5
-        period: day
+    description: dummy_project User
+    config:
+      freshness:
+        warn_after:
+          count: 3
+          period: day
+        error_after:
+          count: 5
+          period: day
+      loaded_at_field: created_at
     columns:
     - name: id
       description: ID
@@ -661,10 +725,11 @@ sources:
       data_type: int64
       data_tests:
       - relationships:
-          to: source('dummy', 'companies')
-          field: id
-          meta:
-            relationship_type: many-to-one
+          arguments:
+            to: source('dummy', 'companies')
+            field: id
+            meta:
+              relationship_type: many-to-one
 
 ```
 
@@ -846,20 +911,12 @@ Example:
 >
 > The output will be as shown below. It is recommended to indent the YAML file with a tool of your choice.
 
-> [!WARNING]
->
-> If you are using a version of dbt lower than v1.8, replace `tests:` with `data_tests:` in the generated file.
->
-> [Add data tests to your DAG | dbt Developer Hub](https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax)
->
->> Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, the key was renamed from `tests:` to `data_tests:`.
-
 ```yaml
 ---
 version: 2
 models:
 - name: stg_dummy__profiles
-  description: Write a logical_name of the 'profiles' table.
+  description: Write the logical_name of the 'profiles' table in 'dummy_project'.
   columns:
   - name: profile_id
     description: profile_id
@@ -868,10 +925,11 @@ models:
     - unique
     - not_null
     - relationships:
-        to: source('dummy', 'profiles')
-        field: id
-        meta:
-          relationship_type: one-to-one
+        arguments:
+          to: source('dummy', 'profiles')
+          field: id
+          meta:
+            relationship_type: one-to-one
   - name: user_id
     description: user_id
     data_type: int64
@@ -879,10 +937,11 @@ models:
     - unique
     - not_null
     - relationships:
-        to: source('dummy', 'users')
-        field: id
-        meta:
-          relationship_type: one-to-one
+        arguments:
+          to: source('dummy', 'users')
+          field: id
+          meta:
+            relationship_type: one-to-one
   - name: first_name
     description: Write a description of the 'profiles.first_name' column.
     data_type: string
@@ -906,7 +965,7 @@ models:
 
 ```
 
-### Generated dbt Seed Files
+### Generate dbt Seed Files
 
 #### dbt Seed Configuration
 
@@ -917,7 +976,7 @@ You can configure `defaults` in this file.
 
 Set the default value for the `description` of the `seeds` enum.
 
-In the `description` of `seed_descriptions.enum`, you can refer to the source name with `{{ source_name }}`, the translated table name with `{{ translated_table_name }}`, and the translated column name with `{{ translated_attribute_name }}`.
+In the `description` of `seed_descriptions.enum`, you can refer to the project name with `{{ project_name }}`, the table logical table name with `{{ table_logical_name }}`, and the column description with `{{ column_description }}`.
 
 Example:
 
@@ -925,7 +984,7 @@ Example:
 defaults:
   seed_descriptions:
     enum:
-      description: "{{ source_name }} {{ translated_table_name }} {{ translated_attribute_name }} enum"
+      description: "{{ project_name }} {{ table_logical_name }} {{ column_description }} enum"
 
 ```
 
@@ -935,7 +994,7 @@ If nothing is set, it defaults to the following:
 defaults:
   seed_descriptions:
     enum:
-      description: "{{ source_name }} {{ translated_table_name }} {{ translated_attribute_name }} enum"
+      description: "{{ project_name }} {{ table_logical_name }} {{ column_description }} enum"
 
 ```
 
@@ -980,47 +1039,43 @@ Example:
 >
 > The output will be as shown below. It is recommended to indent the YAML file with a tool of your choice.
 
-> [!WARNING]
->
-> If you are using a version of dbt lower than v1.8, replace `tests:` with `data_tests:` in the generated file.
->
-> [Add data tests to your DAG | dbt Developer Hub](https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax)
->
->> Data tests were historically called "tests" in dbt as the only form of testing available. With the introduction of unit tests in v1.8, the key was renamed from `tests:` to `data_tests:`.
-
 ```yaml
 ---
 version: 2
 seeds:
 - name: seed_dummy__post_enum_statuses
-  description: dummy Post Status enum
+  description: dummy_project Post Status enum
   config:
     column_types:
       status_before_type_of_cast: int64
       status_key: string
       status_en: string
       status_ja: string
-columns:
-- name: status_before_type_of_cast
-  description: Status
-  data_tests:
-  - unique
-  - not_null
-- name: status_key
-  description: Status(key)
-  data_tests:
-  - unique
-  - not_null
-- name: status_en
-  description: Status(en)
-  data_tests:
-  - unique
-  - not_null
-- name: status_ja
-  description: Status(ja)
-  data_tests:
-  - unique
-  - not_null
+  columns:
+  - name: status_before_type_of_cast
+    description: Status(before_type_of_cast)
+    data_type: int64
+    data_tests:
+    - unique
+    - not_null
+  - name: status_key
+    description: Status(key)
+    data_type: string
+    data_tests:
+    - unique
+    - not_null
+  - name: status_en
+    description: Status(en)
+    data_type: string
+    data_tests:
+    - unique
+    - not_null
+  - name: status_ja
+    description: Status(ja)
+    data_type: string
+    data_tests:
+    - unique
+    - not_null
 
 ```