Standardize unioning macros to use fivetran_utils versions

.gitignore

@@ -61,6 +61,7 @@ service-account.json
 .cache/
 *.temp
 *.tmp
+worktrees/
 
 # Virtual environments
 .conda/

CHANGELOG.md

@@ -1,3 +1,11 @@
+# dbt_github v1.4.0
+
+[PR #80](https://github.com/fivetran/dbt_github/pull/80) includes the following updates:
+
+## Under the Hood
+- Migrates the `union_connections`, `apply_source_relation`, and `partition_by_source_relation` macros to the `dbt_fivetran_utils` package.
+- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_github/#source-casing-for-case-sensitive-destinations) section of the README for details.
+
 # dbt_github v1.3.0
 
 [PR #76](https://github.com/fivetran/dbt_github/pull/76) includes the following updates:
@@ -221,4 +229,4 @@ This release contains the following updates:
   - Please note, if you are installing a version of `dbt_utils` in your `packages.yml` that is not in the range above then you will encounter a package dependency error.
 
 # dbt_github v0.1.0 -> v0.3.0
-Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!
+Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!

README.md

@@ -1,14 +1,14 @@
 <!--section="github_transformation_model"-->
-# Github dbt Package
+# GitHub dbt Package
 
-This dbt package transforms data from Fivetran's Github connector into analytics-ready tables.
+This dbt package transforms data from Fivetran's GitHub connector into analytics-ready tables.
 
 ## Resources
 
 - Number of materialized models¹: 34
 - Connector documentation
-  - [Github connector documentation](https://fivetran.com/docs/connectors/applications/github)
-  - [Github ERD](https://fivetran.com/docs/connectors/applications/github#schemainformation)
+  - [GitHub connector documentation](https://fivetran.com/docs/connectors/applications/github)
+  - [GitHub ERD](https://fivetran.com/docs/connectors/applications/github#schemainformation)
 - dbt package documentation
   - [GitHub repository](https://github.com/fivetran/dbt_github)
   - [dbt Docs](https://fivetran.github.io/dbt_github/#!/overview)
@@ -47,7 +47,7 @@ By default, this package materializes the following final tables:
 ## Prerequisites
 To use this dbt package, you must have the following:
 
-- At least one Fivetran Github connection syncing data into your destination.
+- At least one Fivetran GitHub connection syncing data into your destination.
 - A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.
 
 ## How do I use the dbt package?
@@ -65,7 +65,7 @@ Include the following github package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/github
-    version: [">=1.3.0", "<1.4.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=1.4.0", "<1.5.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/github_source` in your `packages.yml` since this package has been deprecated.
@@ -79,14 +79,12 @@ dispatch:
 ```
 
 ### Define database and schema variables
-
 #### Option A: Single connection
-By default, this package runs using your [destination](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile) and the `github` schema. If this is not where your GitHub data is (for example, if your github schema is named `github_fivetran`), add the following configuration to your root `dbt_project.yml` file:
+By default, this package runs using your destination and the `github` schema. If this is not where your GitHub data is (for example, if your GitHub schema is named `github_fivetran`), add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
 vars:
-  github:
-    github_database: your_database_name
+    github_database: your_destination_name
     github_schema: your_schema_name
 ```
 
@@ -110,42 +108,9 @@ vars:
         name: connection_2_source_name
 ```
 
-##### Recommended: Incorporate unioned sources into DAG
-> *If you are running the package through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore), the below step is necessary in order to synchronize model runs with your GitHub connections. Alternatively, you may choose to run the package through Fivetran [Quickstart](https://fivetran.com/docs/transformations/quickstart), which would create separate sets of models for each GitHub source rather than one set of unioned models.*
-
-By default, this package defines one single-connection source, called `github`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your GitHub sources, though the package will run successfully.
+#### Optional: Incorporate unioned sources into DAG
 
-To properly incorporate all of your GitHub connections into your project's DAG:
-1. Define each of your sources in a `.yml` file in the `models` directory of your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_github.yml` [file](https://github.com/fivetran/dbt_github/blob/main/models/staging/src_github.yml).
-
-```yml
-# a .yml file in your root project
-
-version: 2
-
-sources:
-  - name: <name> # ex: Should match name in github_sources
-    schema: <schema_name>
-    database: <database_name>
-    loader: fivetran
-    config:
-      loaded_at_field: _fivetran_synced
-      freshness: # feel free to adjust to your liking
-        warn_after: {count: 72, period: hour}
-        error_after: {count: 168, period: hour}
-
-    tables: # copy and paste from github/models/staging/src_github.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
-```
-
-> **Note**: If there are source tables you do not have (see [Disable models for non-existent sources](https://github.com/fivetran/dbt_github?tab=readme-ov-file#disable-models-for-non-existent-sources)), you may still include them, as long as you have set the right variables to `False`.
-
-2. Set the `has_defined_sources` variable (scoped to the `github` package) to `True`, like such:
-```yml
-# dbt_project.yml
-vars:
-  github:
-    has_defined_sources: true
-```
+If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple GitHub connections, you can define your sources in a property `.yml` file, [using this as a template](https://github.com/fivetran/dbt_github/blob/main/models/staging/src_github.yml). Set the variable `has_defined_sources: true` under the GitHub namespace in your `dbt_project.yml`. Otherwise, your GitHub connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.
 
 ### Disable models for non-existent sources
 Your GitHub connection might not sync every table that this package expects. If your syncs exclude certain tables, it is because you either don't use that functionality in GitHub or have actively excluded some tables from your syncs.
@@ -186,6 +151,14 @@ If an individual source table has a different name than the package expects, add
 vars:
     github_<default_source_table_name>_identifier: your_table_name 
 ```
+
+#### Source casing for case-sensitive destinations
+By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:
+
+```yml
+vars:
+    fivetran_using_source_casing: true
+```
 </details>
 
 ### (Optional) Orchestrate your models with Fivetran Transformations for dbt Core™

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'github'
-version: '1.3.0'
+version: '1.4.0'
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 models:
   github:

integration_tests/ci/test_scenarios.yml

@@ -24,4 +24,10 @@ test_scenarios:
       github__using_issue_label: false
       github__using_requested_reviewer_history: false
       github__using_label: false
-    include_incremental: false
+    include_incremental: false
+
+  - name: "MDLS"
+    vars:
+      fivetran_using_source_casing: true
+    include_incremental: false
+    include_dbt_seed: true

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'github_integration_tests'
-version: '1.3.0'
+version: '1.4.0'
 config-version: 2
 profile: 'integration_tests'
 vars:
@@ -30,7 +30,7 @@ seeds:
   +docs:
     show: false
   github_integration_tests:
-    +quote_columns: "{{ true if target.type == 'redshift' else false }}"
+    +quote_columns: "{{ target.type == 'redshift' or var('fivetran_using_source_casing', false) }}"
     github_issue_assignee_data:
       +column_types:
         _fivetran_synced: timestamp