[Feature] Supports non half width alphanumeric for generic test

[d-kaneshiro]

resolves #8204

Problem

Double-byte characters in GenericTest were displayed as "_".

Example

Double-byte character columns are displayed as "_" when dbt test is executed.

Solution

Fixed an issue where double-byte characters were not displayed in GenericTest.

Example

All double-byte character columns are displayed when 'dbt test' is executed.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (bb35b3e) 86.54% compared to head (b0a988a) 86.49%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8203      +/-   ##
==========================================
- Coverage   86.54%   86.49%   -0.05%     
==========================================
  Files         179      179              
  Lines       26538    26538              
==========================================
- Hits        22967    22955      -12     
- Misses       3571     3583      +12     

Flag	Coverage Δ
integration	83.34% <100.00%> (-0.11%)	⬇️
unit	64.79% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
core/dbt/parser/generic_test_builders.py	94.00% <100.00%> (ø)

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[d-kaneshiro]

Added tests that include full-width hiragana, katakana, kanji, and half-width katakana columns.
All of these are working as intended.
CI test with GitHubActions has also been successful.
On the other hand, I have not verified the operation in environments other than Japan.

[dbeatty10]

Thanks for raising this PR and adding these test cases @d-kaneshiro !

@gshank and I looked at this, and we'd like to fix it further upstream.

@gshank is going to take a deeper look to identify the best place to address this and get back to you.

[gshank]

d-kaneshiro: Please remove the changes in core/dbt/task/runnable.py and change a line in core/dbt/parser/generic_test_builders.py from "clean_flat_args = [re.sub("[^0-9a-zA-Z_]+", "", arg) for arg in flat_args]" to "clean_flat_args = [re.sub("\W+", "", arg) for arg in flat_args]".

[gshank]

In addition, the tests/adapter set of tests is for tests that might be different in different adapters, and this functionality should not be different for different adapters, so please move the test to tests/functional/schema_tests.

[dbeatty10]

@d-kaneshiro there's three main things that we'd need to do for this PR:

1. Move the test to the tests/functional/schema_tests directory

2. Change a line in core/dbt/parser/generic_test_builders.py from this:

   clean_flat_args = [re.sub("[^0-9a-zA-Z_]+", "", arg) for arg in flat_args]

   to this:

   clean_flat_args = [re.sub("\W+", "", arg) for arg in flat_args]

3. Remove all of your changes in core/dbt/task/runnable.py (since it should be covered by the above change)

Question

Would you be willing to make these changes? Or would you prefer us to make those changes within your branch?

[d-kaneshiro]

@dbeatty10 and @gshank
Thank you for your advice!

@dbeatty10
Feel free to implement these changes in my branch. (Because I can't think of a best practice for incorporating tests without breaking the existing configuration)
Please let me know after you make any changes. I'll check to see if it behaves as intended.

[gshank]

I do think that putting the functional test in the adapter zone isn't necessary, since this is core dbt functionality and not related to adapters. But it's not that big of an issue. I think we might be rearranging some of this soon anyway.