@Micronauttest-annotated tests can hang indefinitely on Java 21

[j1m-renwick]

Expected Behavior

@Micronauttest tests never hang indefinitely

Actual Behaviour

When running our tests via gradle on the ubuntu-latest github runner, the testing can occassionally stop and hang indefinitely. The hanging always occurs after the following line is printed:

[Test worker] INFO io.micronaut.context.DefaultApplicationContext$RuntimeConfiguredEnvironment - Established active environments: [test]

However, the test that the hang occurs on is not consistent at all and changes each run, and sometimes the test suite will complete without incident. I estimate that approximately one in every 5 runs results in an indefinite hang condition.

It seems worth mentioning that some of the tests use Testcontainer Mongo and Redis container, although as mentioned these are not always the tests that the hang occurs on.

Steps To Reproduce

So far I have only been able to reproduce this locally by creating a docker container that approximates the github runner which has, and running it on 1 CPU (to approximate the 2vCPU that the normal github runner runs on for private repos).

# create the container with 1 CPU on ubuntu-latest
docker run -it --privileged --cpus="1" --cpuset-cpus="0" -v /Users/bob/myProject:/mnt/project ubuntu:latest

# Inside the container:

apt-get update
apt-get install -y docker.io wget unzip tar

# Download Java 21 adopt distribution which we specify the runner to use
wget https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.3%2B9/OpenJDK21U-jdk_x64_linux_hotspot_21.0.3_9.tar.gz -O /tmp/OpenJDK21U-jdk_x64_linux_hotspot_21.0.3_9.tar.gz

# Create a directory for the JDK installation
mkdir -p /usr/lib/jvm

# Extract the downloaded tar.gz file
tar -xzf /tmp/OpenJDK21U-jdk_x64_linux_hotspot_21.0.3_9.tar.gz -C /usr/lib/jvm

# Set up environment variables to use this Java version
export JAVA_HOME=/usr/lib/jvm/jdk-21.0.3+9
export PATH=$JAVA_HOME/bin:$PATH

# Add environment variables to the profile to make them persistent
echo "export JAVA_HOME=/usr/lib/jvm/jdk-21.0.3+9" >> /etc/profile.d/jdk21.sh
echo "export PATH=\$JAVA_HOME/bin:\$PATH" >> /etc/profile.d/jdk21.sh

# Source the profile to apply changes immediately
source /etc/profile.d/jdk21.sh

# Verify the installation
java -version

# start docker daemon as a background process
nohup dockerd &

# Verify Docker daemon is running
docker ps

# go to project directory
cd /mnt/project

./gradlew clean build test -i

As mentioned, the last command usually needs to be run several times to get the hang condition..

I took a thread dump of the longest-lived java process when the hanging state occurred:

threaddump.txt

Environment Information

Mentioned in the reproducing steps above for the docker image:
OS - ubuntu-latest( aka noble / 24.04)
JVM - 21.0.3 adopt JDK

Example Application

No response

Version

4.3.1

[graemerocher]

do you have an example app you can attach that reproduces the issue?

[j1m-renwick]

hi @graemerocher - not one that can be shared at the moment unfortunately, but I'll put one together on tuesday when I'm back in the office.

[j1m-renwick]

So I didn't have much joy creating a non-proprietary minimal application to cause the issue, but in experimenting I do seem to have just resolved the issue by setting micronaut.health.monitor.enabled: false in the test yaml. There are a couple of health indicators, one of which is using the TaskExecutors.BLOCKING option and seems like a likely possibility as the issue:

@Singleton
class MyHealthIndicator implements HealthIndicator {

    private final Scheduler scheduler

    @Inject
    MyHealthIndicator(@Named(TaskExecutors.BLOCKING) ExecutorService executorService) {
        this.scheduler = Schedulers.from(executorService)
    }

    @Override
    Publisher<HealthResult> getResult() {
        Flowable.fromCallable{
           // some blocking logic to call a DB
        }
         .subscribeOn(scheduler)
    }

}

I'll see if I can attach an example that includes this next week.

[j1m-renwick]

@graemerocher This project recreates the issue and seems to hang each time it's run inside the docker container mentioned above - normally around the 15th test. The tests pass if the io.micronaut:micronaut-management dependency is removed, or if micronaut.health.monitor.enabled is set as false in the yaml.

the tests occasionally fail with some HealthMonitorTask-related dependency issue, such as the below. But I don't think it's a requirement for this to occur before the tests hang.

17:48:27.243 [scheduled-executor-thread-2] INFO  i.m.c.DefaultApplicationContext$RuntimeConfiguredEnvironment - Established active environments: [test]
    17:48:28.599 [scheduled-executor-thread-2] ERROR i.m.s.DefaultTaskExceptionHandler - Error creating scheduled task for bean [HealthMonitorTask] Failed to inject value for parameter [discoveryClient] of class: io.micronaut.management.health.indicator.discovery.DiscoveryClientHealthIndicator

    Message: No bean of type [io.micronaut.discovery.DiscoveryClient] exists. 
    Path Taken: new HealthMonitorTask(CurrentHealthStatus currentHealthStatus,List healthIndicators) --> new HealthMonitorTask(CurrentHealthStatus currentHealthStatus,[List healthIndicators]) --> new DiscoveryClientHealthIndicator([DiscoveryClient discoveryClient])
    io.micronaut.context.exceptions.DependencyInjectionException: Failed to inject value for parameter [discoveryClient] of class: io.micronaut.management.health.indicator.discovery.DiscoveryClientHealthIndicator

    Message: No bean of type [io.micronaut.discovery.DiscoveryClient] exists. 
    Path Taken: new HealthMonitorTask(CurrentHealthStatus currentHealthStatus,List healthIndicators) --> new HealthMonitorTask(CurrentHealthStatus currentHealthStatus,[List healthIndicators]) --> new DiscoveryClientHealthIndicator([DiscoveryClient discoveryClient])
        at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2181)
        at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1328)
        at io.micronaut.management.health.indicator.discovery.$DiscoveryClientHealthIndicator$Definition.instantiate(Unknown Source)
        at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2330)
        at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2300)
        at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2312)
        at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3114)
        at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80)
        at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:3016)
        at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2977)
        at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2951)
        at io.micronaut.context.DefaultBeanContext.addCandidateToList(DefaultBeanContext.java:3566)
        at io.micronaut.context.DefaultBeanContext.resolveBeanRegistrations(DefaultBeanContext.java:3514)
        at io.micronaut.context.DefaultBeanContext.getBeanRegistrations(DefaultBeanContext.java:3484)
        at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:1451)
        at io.micronaut.context.AbstractBeanResolutionContext.getBeansOfType(AbstractBeanResolutionContext.java:95)
        at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBeansOfType(AbstractInitializableBeanDefinition.java:2253)
        at io.micronaut.context.AbstractInitializableBeanDefinition.getBeansOfTypeForConstructorArgument(AbstractInitializableBeanDefinition.java:1476)
        at io.micronaut.management.health.monitor.$HealthMonitorTask$Definition.instantiate(Unknown Source)
        at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2330)
        at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2300)
        at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2312)
        at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3114)
        at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80)
        at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:3016)
        at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2977)
        at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2751)
        at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1750)
        at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:856)
        at io.micronaut.scheduling.processor.ScheduledMethodProcessor.lambda$process$2(ScheduledMethodProcessor.java:123)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
    Caused by: io.micronaut.context.exceptions.NoSuchBeanException: No bean of type [io.micronaut.discovery.DiscoveryClient] exists. 
        at io.micronaut.context.DefaultBeanContext.newNoSuchBeanException(DefaultBeanContext.java:2787)
        at io.micronaut.context.DefaultApplicationContext.newNoSuchBeanException(DefaultApplicationContext.java:307)
        at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2756)
        at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1750)
        at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:89)
        at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2165)
        ... 35 common frames omitted

Let me know if there's anything else needed, cheers.

test-hang-example.zip

[mmwinther]

We have also experienced this problem:

• Consistently on every build
• Only on Github Actions with ubuntu runner (not locally on mac)
• Only with test-resources enabled
• Only with management endpoints enabled

We found that pinning io.micronaut:micronaut-management:4.5.3 (the latest version at time of writing) resolved the problem.

[graemerocher]

I have not been able to reproduce yet. You can possibly workaround in your tests by setting:

micronaut.health.monitor.enabled=false

In your application-test.properties file.