Correct validation for required fields in variadic groups

[domna]

Fixes

• Correct validation of repeating groups inside the template
• Correct validation of variadic groups with an optional parent

Additions

This adds a Collector class for the validation to collect and log validation problems in one central place. This can be used for a verifier.

Initial problem

Since merging of #310 I get the following error with the mpes example:

LookupError: The data entry, /ENTRY[entry]/INSTRUMENT[instrument]/sourceTYPE[source_probe]/probe,
has an optional parent, /ENTRY[entry]/INSTRUMENT[instrument]/sourceTYPE[sourcetype],
with required children set.
Either provide no children for /ENTRY[entry]/INSTRUMENT[instrument]/sourceTYPE[sourcetype] 
or provide all required ones.

This PR attempts to fix the issue and adds tests for the particular issue

[domna]

@sherjeelshabih The actual problem here is fixed. However, the test do strange things but it's probably that I did something wrong. When I test it with the mpes example it all works as expected now. Nevermind, figured it out

[coveralls]

Pull Request Test Coverage Report for Build 8844173385

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 176 of 186 (94.62%) changed or added relevant lines in 3 files are covered.
• 52 unchanged lines in 4 files lost coverage.
• Overall coverage increased (+0.7%) to 78.622%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
tests/dataconverter/test_helpers.py	28	30	93.33%
pynxtools/dataconverter/helpers.py	142	150	94.67%

Files with Coverage Reduction	New Missed Lines	%
pynxtools/definitions/dev_tools/utils/nxdl_utils.py	1	74.23%
pynxtools/dataconverter/helpers.py	14	90.89%
pynxtools/dataconverter/template.py	15	86.05%
pynxtools/dataconverter/convert.py	22	75.69%

Totals
Change from base Build 8741788106:	0.7%
Covered Lines:	2784
Relevant Lines:	3541

---

💛 - Coveralls

[domna]

We also have a second problem here:

pynxtools/pynxtools/dataconverter/helpers.py

Lines 514 to 516 in 10b1c44

	is_path_in_data_dict, renamed_path = path_in_data_dict(
	nxdl_path, tuple(data.keys())
	)

The renamed_path only contains the generic path. So if we only write a variation of this in the dict and the generic one is still contained in the dict it reports it as required but missing even though a variation is present.

Also the new field name doesn't end up in the required dict in the Template. I have the following idea: When we insert something in the template it checks if the inserted check is a variation of an already existing key. If this is the case it removes the generic group entirely. That way we make sure that 1. we always have the required keys present if nothing is provided and 2. we don't have to deal with the generic keys if a group with another variadic name has been provided. The rest of the implementation should be able to figure everything out based on the variation. What do you think @sherjeelshabih?

[sherjeelshabih]

I like this idea. Please give it a try. We also have this func:

pynxtools/pynxtools/dataconverter/helpers.py

Line 541 in 10b1c44

def try_undocumented(data, nxdl_root: ET.Element):

If somehow we need a complete Template to correctly place such a field, this func can help. But it's another loop so it will be better to avoid this.

[domna]

@sherjeelshabih I got a first version working, which also does the group checking. I still need to clean it up a bit but if you want to have a look already feel free to comment on it already.

Edit: It's cleaned up now and ready to be reviewed

[domna]

I also added a collector class so we can actually check whether a validation was successful or not and bundle the logging in one place. This is needed for #133.

[sanbrock]

LGTM