Set up outputs properly

[andylolz]

Okay so there are three different outputs:

1. Summary of results i.e. pass/fail/not relevant activity counts per scenario
2. Log of reasons for each activity failing each scenario
3. General informational log

We may want to output (1) to stdout or to a file. We probably just want to output the others to files.

We currently solve (1) and (2) using formatters, and (3) using a logger.

[andylolz]

97a59f1 does (1) a bit more nicely.

[andylolz]

Reopening because this really isn’t fixed yet.

[edugomez]

2. Log of reasons for each activity failing each scenario

It would be really nice if instead of scenario_outline.ouput files we get scenario_outline.json. Something like along these lines:

{'errors': [{'message': '2400-03-03 should be on or before today (2017-08-15)',
             'path': '/iati-activities/iati-activity[1]/transaction[2]/value'},
            {'message': '2500-04-04 should be on or before today (2017-08-15)',
             'path': '/iati-activities/iati-activity[1]/transaction[3]/value'},
            {'message': '2600-05-05 should be on or before today (2017-08-15)',
             'path': '/iati-activities/iati-activity[1]/transaction[4]/value'}],
 'id': 'AA-AAA-123123-AA123'}

Also, I think the actual rule applied is only given in the title of the output files, is that right? It would be nice to have the rule (and perhaps the Feature name as in e.g. 'Feature: Elements use a valid format' in elements_use_valid_format.feature file )

And one more thing, would it make sense to have the option of consolidating all the output/json files into a single file, perhaps namespaced by rule applied? This is not essential but I think it would be nice to have a single file to look at and also avoiding the overhead of opening and closing few files?

[andylolz]

As discussed in person: all of this sounds good! Pull request for any or all gratefully received :) (Or if you don’t have time to get to any of it, I’ll certainly have a go!)