Spuriously failing tests #112
Assignees
Comments
|
Full example log of failure. Test: 'equality on inequal arity tuples typing fails' Log |
|
Reference log of succeeding run. Test: 'equality on inequal arity tuples typing fails' Log |
AZWN
added
State-Open
State-Needs-Clarification
Pri-High
and removed
State-Triage
labels
|
More detailed log of a failure. Error message is retained until the end. Apparently, the message gets lost after NaBL2 solving phase, Test: different element types make different tuple types Log |
|
Snippet from build 993. Shows that the message gets lost anywhere between the NaBL2 solver and the SPT runner, so most likely in the NaBL2 runtime. Test: equality on inequal arity tuples typing fails Log |
|
Snippet from build 1002. Test: 'equality on inequal arity tuples typing fails' Log |
|
Excerpt from build 1003. Shows that two resource keys actually put data in the constraint context:
Test: 'equality on inequal arity tuples typing fails ' Log |
|
Excerpt from build 1005. Shows error messages are actually generated, but only in the project-result, not on the file result. Machine: buildfarm Log 'equality on inequal arity tuples typing fails 'Log 'different element types make different tuple types' |
|
In the above entry, there is a snippet In a local (succeeding) build, this part looks as follows: The first statements origin is AbstractConstraintAnalyzer.java#L377, while the second comes from line 206. Expectation is now that the |
|
Detailed Multimap Logging Test: 'different element types make different tuple types' Log |
Diff of (detailed) logs of build 1018 (passing) and 1019 (failing)ObservationsThis diff shows a few interesting things:
ConclusionsI think we can draw a few conclusions from this. First, it is most likely that some non-determinism/bug in the solver is the cause of this behavior. AFAICT now, the constraint(s) giving rise to the error message can be solved in the file-specific phase. For some reason, the solver incidentally postpones some constraints however. More analysis of this behavior is needed. This implies that most likely, the multimap data structure is not the cause of the bug. In addition, the failing JSGLR tests are then maybe unrelated. A competing hypothesis might be that the non-determinism manifests in multiple places, so there is no single cause for this behavior. Currently, I consider that unlikely, as all messages so far can be explained by this hypothesis. Finally, it must be remarked that this behavior, although weird, is not yet a proof of unsoundness. Eventually, an appropriate error is created. Second, I consider the behavior of the SPT evaluator incorrect. For many reasons, an error in an SPT test might be generated in the project-specific phase. There is no valid reason to ignore the messages from the project-phase result. |
|
On build 1024, the following snippet occurs: This has again a different error (also not duplicated), so apparently that is normal |
|
Using 'statix.test; messages: [Cannot' as search string in logs so far (until build 1028) yields no results. |
|
Spoofax 2 build #1029 still fails with the same issue, despite the changes to SPT from a few weeks ago. |
|
In an effort to reproduce Daniels report, I tried the SPT CMD interface. I copied the Not sure what a proper explanation of this is though. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug description
This is a tracking issue for the spurious error message loss in
statix.testVersions
Spoofax version: Nightly
Steps to reproduce the behavior
Run
mvn verifyin/statix.testObserved behavior
Sometimes, a test will fail, e.g.
Expected behavior
No failing test
Additional context
Probably due to #110
Please report instances of failures using the following template:
The text was updated successfully, but these errors were encountered: