Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to run idflakies on following projects #19

Open
byjiang1996 opened this issue Dec 11, 2020 · 2 comments
Open

Failed to run idflakies on following projects #19

byjiang1996 opened this issue Dec 11, 2020 · 2 comments

Comments

@byjiang1996
Copy link
Contributor

The latest testrunner(1.1)/iDFlakies(1.1.0) failed to run iDFlakies on Sentinel repo:

Project: https://github.com/alibaba/Sentinel
Branch: master
SHA: 8c4f251922365029a36ca6fe912b8aa88d6fe286
Module/submodule name: sentinel-demo-quarkus

Error logs:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): com.alibaba.csp:sentinel-demo-quarkus:1.8.1-SNAPSHOT
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (2 tests executed, 4 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelAnnotationQuarkusAdapter(), com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelJaxRsQuarkusAdapter())
at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
at scala.util.Success.flatMap(Try.scala:247)
at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
at edu.illinois.cs.dt.tools.detection.detectors.RandomDetector.(RandomDetector.java:26)
at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:19)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.detectorExecute(DetectorPlugin.java:240)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.lambda$execute$5(DetectorPlugin.java:201)
at edu.illinois.cs.dt.tools.utility.ErrorLogger.runAndLogError(ErrorLogger.java:39)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.execute(DetectorPlugin.java:201)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at edu.illinois.cs.testrunner.mavenplugin.TestPluginPlugin.execute(TestPluginPlugin.scala:102)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
@byjiang1996
Copy link
Contributor Author

In addition, I have minimized the repro steps by modifying .dtfixingtools/original-order file:

If we tested on either com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelAnnotationQuarkusAdapter() or com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelJaxRsQuarkusAdapter() function, similar error will occur:

Only test com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelAnnotationQuarkusAdapter() in original-order file:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): com.alibaba.csp:sentinel-demo-quarkus:1.8.1-SNAPSHOT
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 1 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelAnnotationQuarkusAdapter())
at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
at scala.util.Success.flatMap(Try.scala:247)
at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
at edu.illinois.cs.dt.tools.detection.detectors.OriginalDetector.(OriginalDetector.java:28)
at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:23)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.detectorExecute(DetectorPlugin.java:240)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.lambda$execute$5(DetectorPlugin.java:201)
at edu.illinois.cs.dt.tools.utility.ErrorLogger.runAndLogError(ErrorLogger.java:39)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.execute(DetectorPlugin.java:201)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at edu.illinois.cs.testrunner.mavenplugin.TestPluginPlugin.execute(TestPluginPlugin.scala:102)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

Only test com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelJaxRsQuarkusAdapter() in original-order file:

ERROR (WRITE_ERROR_STDOUT_THROWABLE): com.alibaba.csp:sentinel-demo-quarkus:1.8.1-SNAPSHOT
java.lang.RuntimeException: Set of executed tests is not equal to test list that should have been executed (0 tests executed, 1 tests expected). Did you use testplugin.runner.idempotent.num.runs? Missing tests are: Set(com.alibaba.csp.sentinel.demo.quarkus.NativeGreetingResourceIT#testSentinelJaxRsQuarkusAdapter())
at edu.illinois.cs.testrunner.runner.SmartRunner.$anonfun$run$1(SmartRunner.scala:44)
at scala.util.Success.flatMap(Try.scala:247)
at edu.illinois.cs.testrunner.runner.SmartRunner.run(SmartRunner.scala:36)
at edu.illinois.cs.testrunner.runner.Runner.runList(Runner.scala:28)
at edu.illinois.cs.testrunner.runner.Runner.runList$(Runner.scala:27)
at edu.illinois.cs.testrunner.runner.SmartRunner.runList(SmartRunner.scala:22)
at edu.illinois.cs.dt.tools.detection.DetectorUtil.originalResults(DetectorUtil.java:32)
at edu.illinois.cs.dt.tools.detection.detectors.OriginalDetector.(OriginalDetector.java:28)
at edu.illinois.cs.dt.tools.detection.detectors.DetectorFactory.makeDetector(DetectorFactory.java:23)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.detectorExecute(DetectorPlugin.java:240)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.lambda$execute$5(DetectorPlugin.java:201)
at edu.illinois.cs.dt.tools.utility.ErrorLogger.runAndLogError(ErrorLogger.java:39)
at edu.illinois.cs.dt.tools.detection.DetectorPlugin.execute(DetectorPlugin.java:201)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at edu.illinois.cs.testrunner.mavenplugin.TestPluginPlugin.execute(TestPluginPlugin.scala:102)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

@byjiang1996
Copy link
Contributor Author

The error message in this issue is same as in #22, but the cause may be different as this test class has a project-specific class annotation, while the other case seems to be about parameterized tests.

FYI, similar to #22, all error tests use Junit5 api.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@byjiang1996