[CT-2142] [Regression] dbt 1.5.0 return invalid exit codes.

[elongl]

Is this a regression in a recent version of dbt-core?

• I believe this is a regression in dbt-core functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

Errored models and failed tests return 0.

Expected/Previous Behavior

They should return 1.

Steps To Reproduce

Run a model with a syntax error on dbt 1.5.0+.

Relevant log output

$ dbt run -s error_model
10:27:28  Running with dbt=1.4.1
10:27:29  Found 39 models, 59 tests, 1 snapshot, 0 analyses, 950 macros, 2 operations, 14 seed files, 4 sources, 2 exposures, 0 metrics
10:27:29
10:27:29
10:27:29  Running 1 on-run-start hook
10:27:29  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
10:27:29  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
10:27:29
10:27:29  Concurrency: 255 threads (target='postgres')
10:27:29
10:27:29  1 of 1 START sql view model elon_test.error_model .............................. [RUN]
10:27:29  1 of 1 ERROR creating sql view model elon_test.error_model ..................... [ERROR in 0.04s]
10:27:29
10:27:29  Running 1 on-run-end hook
10:27:29  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
10:27:29  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
10:27:29
10:27:29
10:27:29  Finished running 1 view model, 2 hooks in 0 hours 0 minutes and 0.66 seconds (0.66s).
10:27:29
10:27:29  Completed with 1 error and 0 warnings:
10:27:29
10:27:29  Database Error in model error_model (models/error_model.sql)
10:27:29    syntax error at or near "as"
10:27:29    LINE 4:     select 'a's as string
10:27:29                            ^
10:27:29    compiled Code at ./target/run/elementary_integration_tests/models/error_model.sql
10:27:29
10:27:29  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
$ echo $?
1
$ dbt run -s error_model
10:28:26  Running with dbt=1.5.0-b1
10:28:27  Unable to do partial parsing because of a version mismatch
10:28:29  Found 39 models, 59 tests, 1 snapshot, 0 analyses, 958 macros, 2 operations, 14 seed files, 4 sources, 2 exposures, 0 metrics, 0 groups
10:28:29
10:28:29
10:28:29  Running 1 on-run-start hook
10:28:29  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
10:28:29  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
10:28:29
10:28:29  Concurrency: 255 threads (target='postgres')
10:28:29
10:28:29  1 of 1 START sql view model elon_test.error_model .............................. [RUN]
10:28:29  1 of 1 ERROR creating sql view model elon_test.error_model ..................... [ERROR in 0.04s]
10:28:29
10:28:29  Running 1 on-run-end hook
10:28:30  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
10:28:30  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
10:28:30
10:28:30
10:28:30  Finished running 1 view model, 2 hooks in 0 hours 0 minutes and 0.70 seconds (0.70s).
10:28:30
10:28:30  Completed with 1 error and 0 warnings:
10:28:30
10:28:30  Database Error in model error_model (models/error_model.sql)
10:28:30    syntax error at or near "as"
10:28:30    LINE 4:     select 'a's as string
10:28:30                            ^
10:28:30    compiled Code at ./target/run/elementary_integration_tests/models/error_model.sql
10:28:30
10:28:30  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
$ echo $?
0

Environment

- OS: macOS
- Python: 3.9.6
- dbt (working version): 1.4.1
- dbt (regression version): 1.5.0-b1

Which database adapter are you using with dbt?

postgres

Additional Context

No response

[jtcohen6]

@elongl Thanks for the early beta testing, and for the quick report! I can reproduce this:

$ dbt run
10:49:42  Running with dbt=1.5.0-a1
10:49:42  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 410 macros, 0 operations, 1 seed file, 0 sources, 1 exposure, 0 metrics, 0 groups
10:49:42
10:49:42  Concurrency: 5 threads (target='dev')
10:49:42
10:49:42  1 of 1 START sql view model dbt_jcohen.my_model ................................ [RUN]
10:49:42  1 of 1 ERROR creating sql view model dbt_jcohen.my_model ....................... [ERROR in 0.07s]
10:49:42
10:49:42  Finished running 1 view model in 0 hours 0 minutes and 0.36 seconds (0.36s).
10:49:42
10:49:42  Completed with 1 error and 0 warnings:
10:49:42
10:49:42  Database Error in model my_model (models/my_model.sql)
10:49:42    syntax error at or near ";"
10:49:42    LINE 4:     select asldifjas;dl
10:49:42                                ^
10:49:42    compiled Code at target/run/test/models/my_model.sql
10:49:42
10:49:42  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

$ echo $?
0

I've queued this up with the other bugs that we've been uncovering after the switchover to new CLI / API.

[elongl]

@jtcohen6 Thanks a lot for addressing.
When you're saying new API, you're only referring to the CLI, right?
As in, there's not going to be a stable Python API soon.

[jtcohen6]

@elongl In v1.5, we are planning to release an initial stable Python API, for a subset of dbt-core functionality — programmatic invocations. There's a lot this won't include, but it's a first important step on the way there!

Check out #6356 for some more background & links to relevant issues

[jtcohen6]

Documented behavior: https://docs.getdbt.com/reference/exit-codes

Previous logic for this:

dbt-core/core/dbt/main.py

Lines 128 to 157 in 0c06735

	def main(args=None):
	# Logbook warnings are ignored so we don't have to fork logbook to support python 3.10.
	# This _only_ works for regular cli invocations.
	warnings.filterwarnings("ignore", category=DeprecationWarning, module="logbook")
	if args is None:
	args = sys.argv[1:]
	with log_manager.applicationbound():
	try:
	results, succeeded = handle_and_check(args)
	if succeeded:
	exit_code = ExitCodes.Success.value
	else:
	exit_code = ExitCodes.ModelError.value
	except KeyboardInterrupt:
	# if the logger isn't configured yet, it will use the default logger
	fire_event(MainKeyboardInterrupt())
	exit_code = ExitCodes.UnhandledError.value
	# This can be thrown by eg. argparse
	except SystemExit as e:
	exit_code = e.code
	except BaseException as e:
	fire_event(MainEncounteredError(exc=str(e)))
	if not isinstance(e, dbtException):
	fire_event(MainStackTrace(stack_trace=traceback.format_exc()))
	exit_code = ExitCodes.UnhandledError.value
	sys.exit(exit_code)

Potential way of handling this in click: https://click.palletsprojects.com/en/8.1.x/api/#click.MultiCommand.result_callback

This resolves the issue for me locally, though it doesn't account for "unhandled" errors (= exit code 2):

def cli(ctx, **kwargs):
    """An ELT tool for managing your SQL transformations and data models.
    For more documentation on these commands, visit: docs.getdbt.com
    """

@cli.result_callback()
def process_result(command_result: Tuple[Any, bool], **kwargs):
    results, success = command_result
    if success:
        exit_code = ExitCodes.Success.value    # 0
    else:
        exit_code = ExitCodes.ModelError.value # 1

    ctx = click.get_current_context()
    ctx.exit(code=exit_code)
    
    # not accounted for here:
    # ExitCodes.UnhandledError.value (=2) if KeyboardInterrupt/click.Abort or BaseException

[github-christophe-oudar]

Thanks for reporting the issue!
I tried dbt 1.5 today and I'm reverting because the our CI validation is using exit code to figure out if it's successful.

I use exit_code != 0 condition especially for exit code 1 but abort could happen (for instance, in a graceful shutdown of the container running it) so supporting that is clearly still relevant in my opinion.