SV2V QA test suite

[chili-chips-ba]

While trying SV2V, we quickly realized how good it was compared to both Synlig and Yosys-Slang.

Yet, having run into some here-and-there issues, we believe that the SV2V project would benefit from adding our two SV RTL repos to its regression tests:

• openCam-SI
• opl3_fpga

[zachjs]

Although sv2v has extensive test coverage (you can check its code coverage with make coverage), this repository doesn't include regression tests, per se. sv-tests already covers a handful of large open source SystemVerilog projects. My gut feeling is that it would be best for the community to focus our regression testing efforts into the sv-tests project. What do you think?

[chili-chips-ba]

While we are not sure about the internal workings of sv2v QA process, we were under the impression that you were taking only occasional and cursory look at sv-tests

"... indeed sv2v has hundreds of its own tests, generally focusing on just a few features at a time. I do use the sv-tests dashboard from time to time to check in on its assessment of sv2v..."

We also agree that moving the focus of your QA process to sv-tests is a worthy idea. However, to get the most out of it, the sv-tests would need to be divided into two sub-categories:

• for RTL and Synth
• for Sim and DV, perhaps even with UVM sub-sub-category

or provide a mechanism for marking selected SV features (along with tests that validate them) as NA. Those should then show greyed out and not contribute to your total score. We assume that all that the sv2v cares about at the moment is the RTL/Synth subset of the language.

In the meantime, the life goes on, and your private library of test designs may need to grow with it 😉...

[zachjs]

We also agree that moving the focus of your QA process to sv-tests is a worthy idea. However, to get the most out of it, the sv-tests would need to be divided into two sub-categories:

• for RTL and Synth
• for Sim and DV, perhaps even with UVM sub-sub-category

or provide a mechanism for marking selected SV features (along with tests that validate them) as NA. Those should then show greyed out and not contribute to your total score. We assume that all that the sv2v cares about at the moment is the RTL/Synth subset of the language.

sv-tests provides some value to me even though it complains about features that are out of scope for sv2v. I know these features are out of scope and can focus my efforts elsewhere. Certainly it would be great for sv-tests to incorporate your suggestions to improve its signal to noise ratio, but I don't have the bandwidth to take on that project.

In the meantime, the life goes on, and your private library of test designs may need to grow with it 😉...

I wouldn't call the tests private! They're all committed into the repo. I'm happy for contributors to add more test coverage in line with the guidelines (https://github.com/zachjs/sv2v/blob/master/test/README.md). Indeed, some of the tests come directly from outside contributors or are based on test cases provided in bug reports. If you find a bug in sv2v, please report that bug!

[chili-chips-ba]

@hzeller anything you can do about dividing sv-tests into two sub-categories (1) RTL/Synth; (2) DV/Sim, possibly with UVM as a sub-sub-category?!

[sifferman]

@chili-chips-ba That is how sv-tests currently works for Yosys and Synlig: chipsalliance/sv-tests#5139. The functionality can be easily extended to sv2v

[zachjs]

Thank you for starting this discussion! I don't think there are outstanding action items for sv2v, as sv-tests is better suited for these sorts of large and widely-applicable test cases. I am closing this issue.