GTest test results plugin updates & fixes

- Added mutex for thread safety in post test executable event handler
- Fixed double test failure count reporting
- Added header to distinguish multiple sets of test results for different contexts
- Added test executable run time to GTest output report in appropriate spot
- Made documentation more better

plugins/stdout_gtestlike_tests_report/README.md

@@ -1,18 +1,85 @@
-ceedling-stdout-gtestlike-tests-report
-======================
+# Ceedling Plugin: GTest-like Tests Report
 
-## Overview
+# Overview
 
-The stdout_gtestlike_tests_report replaces the normal ceedling "pretty" output with 
-a variant that resembles the output of gtest. This is most helpful when trying to 
-integrate into an IDE or CI that is meant to work with google test.
+This plugin is intended to be used in place of the more commonly used "pretty" 
+test report plugin. Like its sibling, this plugin ollects raw test results from
+the individual test executables of your test suite and presents them in a more 
+readable summary form — specifically the GoogleTest format.
 
-## Setup
+This plugin is most useful when using an IDE or working with a CI system that
+understands the GTest console logging format.
+
+# Output (Snippet)
+
+## GoogleTest reporting elements
+
+Ceedling's conventions and output map to GTest format as the following:
+
+* A Ceedling test file — ultimately an individual test executable — is a GTest 
+  _test case_.
+* A Ceedling test case (a.k.a. unit test) is a GTest _test_.
+* Execution time is collected for each Ceedling test executable, not each 
+  Ceedling test case. As such, the test report includes only execution time for
+  each GTest _test case_. Individual test execution times are reported as 0 ms.
+
+GoogleTest generates reporting output incrementally. Ceedling produces test 
+results incrementally as well, but its plugin reporting structure does not 
+collect and format statistics until the end of a build. This plugin duplicates
+the tense of the wording in a GTest report, but it is unintentionally somewhat 
+misleading.
+
+## Example output (snippet)
+
+The GTest format is verbose. It lists all tests with success and failure results.
+
+The example output below shows the header and footer of test results for a suite 
+of 49 Ceedling tests in 18 test files but only includes logging for 6 tests.
+
+```
+[==========] Running 49 tests from 18 test cases.
+[----------] Global test environment set-up.
+ 
+ ...
+
+[----------] 4 tests from test/TestUsartModel.c
+[ RUN      ] test/TestUsartModel.c.testGetBaudRateRegisterSettingShouldReturnAppropriateBaudRateRegisterSetting
+[       OK ] test/TestUsartModel.c.testGetBaudRateRegisterSettingShouldReturnAppropriateBaudRateRegisterSetting (0 ms)
+[ RUN      ] test/TestUsartModel.c.testGetFormattedTemperatureFormatsTemperatureFromCalculatorAppropriately
+[       OK ] test/TestUsartModel.c.testGetFormattedTemperatureFormatsTemperatureFromCalculatorAppropriately (0 ms)
+[ RUN      ] test/TestUsartModel.c.testShouldReturnErrorMessageUponInvalidTemperatureValue
+[       OK ] test/TestUsartModel.c.testShouldReturnErrorMessageUponInvalidTemperatureValue (0 ms)
+[ RUN      ] test/TestUsartModel.c.testShouldReturnWakeupMessage
+[       OK ] test/TestUsartModel.c.testShouldReturnWakeupMessage (0 ms)
+[----------] 4 tests from test/TestUsartModel.c (1277 ms total)
+[----------] 1 tests from test/TestMain.c
+[ RUN      ] test/TestMain.c.testMainShouldCallExecutorInitAndContinueToCallExecutorRunUntilHalted
+[       OK ] test/TestMain.c.testMainShouldCallExecutorInitAndContinueToCallExecutorRunUntilHalted (0 ms)
+[----------] 1 tests from test/TestMain.c (1351 ms total)
+[----------] 1 tests from test/TestModel.c
+[ RUN      ] test/TestModel.c.testInitShouldCallSchedulerAndTemperatureFilterInit
+test/TestModel.c(21): error: Function TaskScheduler_Init() called more times than expected.
+ Actual:   FALSE
+ Expected: TRUE
+[  FAILED  ] test/TestModel.c.testInitShouldCallSchedulerAndTemperatureFilterInit (0 ms)
+[----------] 1 tests from test/TestModel.c (581 ms total)
+
+[----------] Global test environment tear-down.
+[==========] 49 tests from 18 test cases ran.
+[  PASSED  ] 48 tests.
+[  FAILED  ] 1 tests, listed below:
+[  FAILED  ] test/TestModel.c.testInitShouldCallSchedulerAndTemperatureFilterInit
+
+ 1 FAILED TESTS
+```
+
+# Configuration
 
 Enable the plugin in your project.yml by adding `stdout_gtestlike_tests_report`
-to the list of enabled plugins.
+to the list of enabled plugins instead of any other `stdout_*_tests_report` 
+plugin.
 
-``` YAML
+```YAML
 :plugins:
   :enabled:
     - stdout_gtestlike_tests_report

plugins/stdout_gtestlike_tests_report/assets/template.erb

@@ -1,8 +1,7 @@
 % ignored        = hash[:results][:counts][:ignored]
 % failed         = hash[:results][:counts][:failed]
 % stdout_count   = hash[:results][:counts][:stdout]
-% header_prepend = ((hash[:header].length > 0) ? "#{hash[:header]}: " : '')
-% banner_width   = 25 + header_prepend.length # widest message
+% name_banner    = (" #{hash[:header]}" + (' ' * (9 - hash[:header].length)))[0..9]
 % results = {}
 % hash[:results][:successes].each do |testresult|
 %   results[ testresult[:source][:file] ] = testresult[:collection]
@@ -28,7 +27,8 @@
 %   end
 % end
 
-
+[==========]
+[<%=name_banner%>]
 [==========] Running <%=hash[:results][:counts][:total].to_s%> tests from <%=results.length.to_s%> test cases.
 [----------] Global test environment set-up.
 % results.each_pair do |modulename, moduledetails|
@@ -57,17 +57,16 @@
 [       OK ] <%=modulename%>.<%=item[:test]%> (0 ms)
 %     end
 %   end
-[----------] <%=moduledetails.length.to_s%> tests from <%=modulename%> (0 ms total)
+%   duration = ((hash[:results][:times][modulename]) * 1000.0).round(0)
+[----------] <%=moduledetails.length.to_s%> tests from <%=modulename%> (<%=duration%> ms total)
 % end
 
 % if (hash[:results][:counts][:total] > 0)
 [----------] Global test environment tear-down.
-[==========] <%=hash[:results][:counts][:total].to_s%> tests from <%=hash[:results][:stdout].length.to_s%> test cases ran.
+[==========] <%=hash[:results][:counts][:total].to_s%> tests from <%=results.length.to_s%> test cases ran.
 [  PASSED  ] <%=hash[:results][:counts][:passed].to_s%> tests.
 %   if (failed == 0)
 [  FAILED  ] 0 tests.
-
- 0 FAILED TESTS
 %   else
 [  FAILED  ] <%=failed.to_s%> tests, listed below:
 %     hash[:results][:failures].each do |failure|

plugins/stdout_gtestlike_tests_report/lib/stdout_gtestlike_tests_report.rb

@@ -5,39 +5,58 @@ class StdoutGtestlikeTestsReport < Plugin
 
   def setup
     @result_list = []
+    @mutex = Mutex.new
+
+    # Fetch the test results template for this plugin
     @plugin_root = File.expand_path(File.join(File.dirname(__FILE__), '..'))
     template = @ceedling[:file_wrapper].read(File.join(@plugin_root, 'assets/template.erb'))
+
+    # Set the report template
     @ceedling[:plugin_reportinator].register_test_results_template( template )
   end
 
+  # Collect result file paths after each test fixture execution
   def post_test_fixture_execute(arg_hash)
-    return if not (arg_hash[:context] == TEST_SYM)
-
-    @result_list << arg_hash[:result_file]
+    # Thread-safe manipulation since test fixtures can be run in child processes
+    # spawned within multiple test execution threads.
+    @mutex.synchronize do
+      @result_list << arg_hash[:result_file]
+    end
   end
 
-  def post_build
+  # Render a report immediately upon build completion (that invoked tests)
+  def post_build()
+    # Ensure a test task was invoked as part of the build
     return if not (@ceedling[:task_invoker].test_invoked?)
 
-    results = @ceedling[:plugin_reportinator].assemble_test_results(@result_list)
+    results = @ceedling[:plugin_reportinator].assemble_test_results( @result_list )
     hash = {
-      :header => '',
+      :header => TEST_SYM.upcase(),
       :results => results
     }
 
     @ceedling[:plugin_reportinator].run_test_results_report(hash)
   end
 
-  def summary
-    result_list = @ceedling[:file_path_utils].form_pass_results_filelist( PROJECT_TEST_RESULTS_PATH, COLLECTION_ALL_TESTS )
+  # Render a test results report on demand using results from a previous build
+  def summary()
+    # Build up the list of passing results from all tests
+    result_list = @ceedling[:file_path_utils].form_pass_results_filelist(
+      PROJECT_TEST_RESULTS_PATH,
+      COLLECTION_ALL_TESTS
+    )
 
-    # get test results for only those tests in our configuration and of those only tests with results on disk
     hash = {
-      :header => '',
-      :results => @ceedling[:plugin_reportinator].assemble_test_results(result_list, {:boom => false})
+      :header => TEST_SYM.upcase(),
+      # Collect all existing test results (success or failing) in the filesystem,
+      # limited to our test collection
+      :results => @ceedling[:plugin_reportinator].assemble_test_results(
+        result_list,
+        {:boom => false}
+      )
     }
 
-    @ceedling[:plugin_reportinator].run_test_results_report(hash)
+    @ceedling[:plugin_reportinator].run_test_results_report( hash )
   end
 
 end