From d7827c08b4cd4abb7609ac89178cb329095b6524 Mon Sep 17 00:00:00 2001 From: Mike Karlesky Date: Thu, 4 Jan 2024 16:10:13 -0500 Subject: [PATCH] GTest test results plugin updates & fixes - Added mutex for thread safety in post test executable event handler - Fixed double test failure count reporting - Added header to distinguish multiple sets of test results for different contexts - Added test executable run time to GTest output report in appropriate spot - Made documentation more better --- .../stdout_gtestlike_tests_report/README.md | 85 +++++++++++++++++-- .../assets/template.erb | 13 ++- .../assets/template.erb copy | 59 ------------- .../lib/stdout_gtestlike_tests_report.rb | 43 +++++++--- 4 files changed, 113 insertions(+), 87 deletions(-) delete mode 100644 plugins/stdout_gtestlike_tests_report/assets/template.erb copy diff --git a/plugins/stdout_gtestlike_tests_report/README.md b/plugins/stdout_gtestlike_tests_report/README.md index 9ab60847..1bcb5678 100644 --- a/plugins/stdout_gtestlike_tests_report/README.md +++ b/plugins/stdout_gtestlike_tests_report/README.md @@ -1,18 +1,85 @@ -ceedling-stdout-gtestlike-tests-report -====================== +# Ceedling Plugin: GTest-like Tests Report -## Overview +# Overview -The stdout_gtestlike_tests_report replaces the normal ceedling "pretty" output with -a variant that resembles the output of gtest. This is most helpful when trying to -integrate into an IDE or CI that is meant to work with google test. +This plugin is intended to be used in place of the more commonly used "pretty" +test report plugin. Like its sibling, this plugin ollects raw test results from +the individual test executables of your test suite and presents them in a more +readable summary form — specifically the GoogleTest format. -## Setup +This plugin is most useful when using an IDE or working with a CI system that +understands the GTest console logging format. + +# Output (Snippet) + +## GoogleTest reporting elements + +Ceedling's conventions and output map to GTest format as the following: + +* A Ceedling test file — ultimately an individual test executable — is a GTest + _test case_. +* A Ceedling test case (a.k.a. unit test) is a GTest _test_. +* Execution time is collected for each Ceedling test executable, not each + Ceedling test case. As such, the test report includes only execution time for + each GTest _test case_. Individual test execution times are reported as 0 ms. + +GoogleTest generates reporting output incrementally. Ceedling produces test +results incrementally as well, but its plugin reporting structure does not +collect and format statistics until the end of a build. This plugin duplicates +the tense of the wording in a GTest report, but it is unintentionally somewhat +misleading. + +## Example output (snippet) + +The GTest format is verbose. It lists all tests with success and failure results. + +The example output below shows the header and footer of test results for a suite +of 49 Ceedling tests in 18 test files but only includes logging for 6 tests. + +``` +[==========] Running 49 tests from 18 test cases. +[----------] Global test environment set-up. + + ... + +[----------] 4 tests from test/TestUsartModel.c +[ RUN ] test/TestUsartModel.c.testGetBaudRateRegisterSettingShouldReturnAppropriateBaudRateRegisterSetting +[ OK ] test/TestUsartModel.c.testGetBaudRateRegisterSettingShouldReturnAppropriateBaudRateRegisterSetting (0 ms) +[ RUN ] test/TestUsartModel.c.testGetFormattedTemperatureFormatsTemperatureFromCalculatorAppropriately +[ OK ] test/TestUsartModel.c.testGetFormattedTemperatureFormatsTemperatureFromCalculatorAppropriately (0 ms) +[ RUN ] test/TestUsartModel.c.testShouldReturnErrorMessageUponInvalidTemperatureValue +[ OK ] test/TestUsartModel.c.testShouldReturnErrorMessageUponInvalidTemperatureValue (0 ms) +[ RUN ] test/TestUsartModel.c.testShouldReturnWakeupMessage +[ OK ] test/TestUsartModel.c.testShouldReturnWakeupMessage (0 ms) +[----------] 4 tests from test/TestUsartModel.c (1277 ms total) +[----------] 1 tests from test/TestMain.c +[ RUN ] test/TestMain.c.testMainShouldCallExecutorInitAndContinueToCallExecutorRunUntilHalted +[ OK ] test/TestMain.c.testMainShouldCallExecutorInitAndContinueToCallExecutorRunUntilHalted (0 ms) +[----------] 1 tests from test/TestMain.c (1351 ms total) +[----------] 1 tests from test/TestModel.c +[ RUN ] test/TestModel.c.testInitShouldCallSchedulerAndTemperatureFilterInit +test/TestModel.c(21): error: Function TaskScheduler_Init() called more times than expected. + Actual: FALSE + Expected: TRUE +[ FAILED ] test/TestModel.c.testInitShouldCallSchedulerAndTemperatureFilterInit (0 ms) +[----------] 1 tests from test/TestModel.c (581 ms total) + +[----------] Global test environment tear-down. +[==========] 49 tests from 18 test cases ran. +[ PASSED ] 48 tests. +[ FAILED ] 1 tests, listed below: +[ FAILED ] test/TestModel.c.testInitShouldCallSchedulerAndTemperatureFilterInit + + 1 FAILED TESTS +``` + +# Configuration Enable the plugin in your project.yml by adding `stdout_gtestlike_tests_report` -to the list of enabled plugins. +to the list of enabled plugins instead of any other `stdout_*_tests_report` +plugin. -``` YAML +```YAML :plugins: :enabled: - stdout_gtestlike_tests_report diff --git a/plugins/stdout_gtestlike_tests_report/assets/template.erb b/plugins/stdout_gtestlike_tests_report/assets/template.erb index fb8e3b13..b312cdb1 100644 --- a/plugins/stdout_gtestlike_tests_report/assets/template.erb +++ b/plugins/stdout_gtestlike_tests_report/assets/template.erb @@ -1,8 +1,7 @@ % ignored = hash[:results][:counts][:ignored] % failed = hash[:results][:counts][:failed] % stdout_count = hash[:results][:counts][:stdout] -% header_prepend = ((hash[:header].length > 0) ? "#{hash[:header]}: " : '') -% banner_width = 25 + header_prepend.length # widest message +% name_banner = (" #{hash[:header]}" + (' ' * (9 - hash[:header].length)))[0..9] % results = {} % hash[:results][:successes].each do |testresult| % results[ testresult[:source][:file] ] = testresult[:collection] @@ -28,7 +27,8 @@ % end % end - +[==========] +[<%=name_banner%>] [==========] Running <%=hash[:results][:counts][:total].to_s%> tests from <%=results.length.to_s%> test cases. [----------] Global test environment set-up. % results.each_pair do |modulename, moduledetails| @@ -57,17 +57,16 @@ [ OK ] <%=modulename%>.<%=item[:test]%> (0 ms) % end % end -[----------] <%=moduledetails.length.to_s%> tests from <%=modulename%> (0 ms total) +% duration = ((hash[:results][:times][modulename]) * 1000.0).round(0) +[----------] <%=moduledetails.length.to_s%> tests from <%=modulename%> (<%=duration%> ms total) % end % if (hash[:results][:counts][:total] > 0) [----------] Global test environment tear-down. -[==========] <%=hash[:results][:counts][:total].to_s%> tests from <%=hash[:results][:stdout].length.to_s%> test cases ran. +[==========] <%=hash[:results][:counts][:total].to_s%> tests from <%=results.length.to_s%> test cases ran. [ PASSED ] <%=hash[:results][:counts][:passed].to_s%> tests. % if (failed == 0) [ FAILED ] 0 tests. - - 0 FAILED TESTS % else [ FAILED ] <%=failed.to_s%> tests, listed below: % hash[:results][:failures].each do |failure| diff --git a/plugins/stdout_gtestlike_tests_report/assets/template.erb copy b/plugins/stdout_gtestlike_tests_report/assets/template.erb copy deleted file mode 100644 index a90f495e..00000000 --- a/plugins/stdout_gtestlike_tests_report/assets/template.erb copy +++ /dev/null @@ -1,59 +0,0 @@ -% ignored = hash[:results][:counts][:ignored] -% failed = hash[:results][:counts][:failed] -% stdout_count = hash[:results][:counts][:stdout] -% header_prepend = ((hash[:header].length > 0) ? "#{hash[:header]}: " : '') -% banner_width = 25 + header_prepend.length # widest message - - -% if (stdout_count > 0) -[==========] Running <%=hash[:results][:counts][:total].to_s%> tests from <%=hash[:results][:stdout].length.to_s%> test cases. -[----------] Global test environment set-up. -% end -% if (failed > 0) -% hash[:results][:failures].each do |failure| -[----------] <%=failure[:collection].length.to_s%> tests from <%=failure[:source][:file]%> -% failure[:collection].each do |item| -[ RUN ] <%=failure[:source][:file]%>.<%=item[:test]%> -% if (not item[:message].empty?) -<%=failure[:source][:file]%>(<%=item[:line]%>): error: <%=item[:message]%> - -% m = item[:message].match(/Expected\s+(.*)\s+Was\s+([^\.]*)\./) -% if m.nil? - Actual: FALSE - Expected: TRUE -% else - Actual: <%=m[2]%> - Expected: <%=m[1]%> -% end -% else -<%=failure[:source][:file]%>(<%=item[:line]%>): fail: <%=item[:message]%> - Actual: FALSE - Expected: TRUE -% end -[ FAILED ] <%=failure[:source][:file]%>.<%=item[:test]%> (0 ms) -% end -[----------] <%=failure[:collection].length.to_s%> tests from <%=failure[:source][:file]%> (0 ms total) -% end -% end -% if (hash[:results][:counts][:total] > 0) -[----------] Global test environment tear-down. -[==========] <%=hash[:results][:counts][:total].to_s%> tests from <%=hash[:results][:stdout].length.to_s%> test cases ran. -[ PASSED ] <%=hash[:results][:counts][:passed].to_s%> tests. -% if (failed == 0) -[ FAILED ] 0 tests. - - 0 FAILED TESTS -% else -[ FAILED ] <%=failed.to_s%> tests, listed below: -% hash[:results][:failures].each do |failure| -% failure[:collection].each do |item| -[ FAILED ] <%=failure[:source][:file]%>.<%=item[:test]%> -% end -% end - - <%=failed.to_s%> FAILED TESTS -% end -% else - -No tests executed. -% end diff --git a/plugins/stdout_gtestlike_tests_report/lib/stdout_gtestlike_tests_report.rb b/plugins/stdout_gtestlike_tests_report/lib/stdout_gtestlike_tests_report.rb index a51438a3..81cf635e 100644 --- a/plugins/stdout_gtestlike_tests_report/lib/stdout_gtestlike_tests_report.rb +++ b/plugins/stdout_gtestlike_tests_report/lib/stdout_gtestlike_tests_report.rb @@ -5,39 +5,58 @@ class StdoutGtestlikeTestsReport < Plugin def setup @result_list = [] + @mutex = Mutex.new + + # Fetch the test results template for this plugin @plugin_root = File.expand_path(File.join(File.dirname(__FILE__), '..')) template = @ceedling[:file_wrapper].read(File.join(@plugin_root, 'assets/template.erb')) + + # Set the report template @ceedling[:plugin_reportinator].register_test_results_template( template ) end + # Collect result file paths after each test fixture execution def post_test_fixture_execute(arg_hash) - return if not (arg_hash[:context] == TEST_SYM) - - @result_list << arg_hash[:result_file] + # Thread-safe manipulation since test fixtures can be run in child processes + # spawned within multiple test execution threads. + @mutex.synchronize do + @result_list << arg_hash[:result_file] + end end - def post_build + # Render a report immediately upon build completion (that invoked tests) + def post_build() + # Ensure a test task was invoked as part of the build return if not (@ceedling[:task_invoker].test_invoked?) - results = @ceedling[:plugin_reportinator].assemble_test_results(@result_list) + results = @ceedling[:plugin_reportinator].assemble_test_results( @result_list ) hash = { - :header => '', + :header => TEST_SYM.upcase(), :results => results } @ceedling[:plugin_reportinator].run_test_results_report(hash) end - def summary - result_list = @ceedling[:file_path_utils].form_pass_results_filelist( PROJECT_TEST_RESULTS_PATH, COLLECTION_ALL_TESTS ) + # Render a test results report on demand using results from a previous build + def summary() + # Build up the list of passing results from all tests + result_list = @ceedling[:file_path_utils].form_pass_results_filelist( + PROJECT_TEST_RESULTS_PATH, + COLLECTION_ALL_TESTS + ) - # get test results for only those tests in our configuration and of those only tests with results on disk hash = { - :header => '', - :results => @ceedling[:plugin_reportinator].assemble_test_results(result_list, {:boom => false}) + :header => TEST_SYM.upcase(), + # Collect all existing test results (success or failing) in the filesystem, + # limited to our test collection + :results => @ceedling[:plugin_reportinator].assemble_test_results( + result_list, + {:boom => false} + ) } - @ceedling[:plugin_reportinator].run_test_results_report(hash) + @ceedling[:plugin_reportinator].run_test_results_report( hash ) end end