NoSuchMethodError Health.down(java.lang.Exception) exception thrown on actuator /health request

[massivespace]

After upgrading to spring boot 3.3.0, if my Kafka brokers are misconfigured (testing), I'm getting an exception in cloud stream on requests to my /actuator/health endpoint:

jakarta.servlet.ServletException: Handler dispatch failed: java.lang.NoSuchMethodError: 'org.springframework.boot.actuate.health.Health$Builder org.springframework.boot.actuate.health.Health.down(java.lang.Exception)'
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1104) ~[spring-webmvc-6.1.8.jar:6.1.8]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.1.8.jar:6.1.8]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.1.8.jar:6.1.8]
...
Caused by: java.lang.NoSuchMethodError: 'org.springframework.boot.actuate.health.Health$Builder org.springframework.boot.actuate.health.Health.down(java.lang.Exception)'
	at org.springframework.cloud.stream.binder.kafka.common.AbstractKafkaBinderHealthIndicator.safelyBuildTopicsHealth(AbstractKafkaBinderHealthIndicator.java:110) ~[spring-cloud-stream-binder-kafka-core-4.1.2.jar:4.1.2]
	at org.springframework.cloud.stream.binder.kafka.common.AbstractKafkaBinderHealthIndicator.doHealthCheck(AbstractKafkaBinderHealthIndicator.java:92) ~[spring-cloud-stream-binder-kafka-core-4.1.2.jar:4.1.2]
	at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:94) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:47) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getLoggedHealth(HealthEndpointSupport.java:172) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:145) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateContribution(HealthEndpointSupport.java:156) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:141) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateContribution(HealthEndpointSupport.java:156) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:141) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:110) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:81) ~[spring-boot-actuator-3.3.0.jar:3.3.0]...
	at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:80) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:69) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:281) ~[spring-core-6.1.8.jar:6.1.8]
	at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74) ~[spring-boot-actuator-3.3.0.jar:3.3.0]
	at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60) ~[spring-boot-actuator-3.3.0.jar:3.3.0]

My pom file is configured with the following package versions:
java : 17
kotlin : 2.0.0
spring-boot-starter-parent : 3.3.0
spring-boot-starter-actuator : [same as parent]
spring-cloud-config-client : 4.1.2
spring-cloud-starter-stream-kafka : 4.1.2
spring-cloud-dependencies : 2023.0.2 (I also tried 2023.0.1)

I verified this works as expected in spring boot 3.2.6.

I don't seem to get the exception during normal operation (properly configured brokers) for a Health.up() call, so it seems related to Health.down specifically with a passed exception. Stepping into the code during the exception, it seems to be lost, as if a dependency is being misinterpreted with a version different from what I'm compiling against.

[sobychacko]

@massivespace Could you elaborate on what you mean by misconfigured broker? Is there a way for us to reproduce the issue? Thanks!

[massivespace]

@massivespace Could you elaborate on what you mean by misconfigured broker? Is there a way for us to reproduce the issue? Thanks!

Sorry about that. Yes, I'm setting the broker via the properties file (application.yml) as follows:
spring.cloud.stream.kafka.binder.brokers: broker_hostname

Under normal circumstances, broker_hostname responds, and all is well. If I just make up the broker hostname (add some junk characters), to test a misconfiguration (i.e. unable to communicate with the broker), that's when it throws the exception.

Other configuration that is applicable:
management.health.livenessstate.enabled: true
management.health.readinessstate.enabled: true
management.endpoint.health.probes.enabled: true
management.endpoint.health.show-details: always
management.health.endpoints.web.exposure.include: health
We also have some legacy settings, don't think they're related, but just in case:
spring.main.allow-bean-definition-overriding: true
spring.main.allow-circular-references: true

I will try to put together a barebones app to see if I can make a reproducer to link here.

[olegz]

I find it very strange as I have just checked that the method is there and was not removed by boot.
Can you check if somehow your real spring-boot-actuator version is different. For example some transitive dependency pulled a different version?

[morskoioleg]

Hello!
I face the same problem with spring-boot-actuator:3.3.1 with same conditions
I double checked my dependencies using gradle dependencies and it is 3.3.1 everywhere.

UPD:
I also found out that this happens only if you set spring.cloud.stream.kafka.binder.brokers with non-resolvable DNS name value.
With correct value, but with wrong port (for example) everything works as expected - I've got DOWN status in /health response.

[sobychacko]

@morskoioleg @massivespace Sorry for the delay on this issue. Trying to get back to it. Could you debug and see what happens when you get to AbstractKafkaBinderHealthIndicator.safelyBuildTopicsHealth? I am curious to know what happens when you get on that line.

What is the easiest way to reproduce the issue on our end? I created a sample application and started it without running a Kafka instance. When invoking the health endpoint, I get the proper DOWN status and do not see the exceptions you encountered. So, I am looking for an easy way to reproduce it.

[RenatMir]

Hello I faced the same problem. initMetadataConsumer() in the AbstractKafkaBinderHealthIndicator.buildTopicsHealth is failing. It is trying to create a consumer But throwing org.apache.kafka.common.KafkaException with message: "No resolvable bootstrap urls given in bootstrap.servers". I use "local.host:9092" for testing purposes. I see, that this method is inside try-catch block which should simply return Health:

catch (Exception ex) { return Health.down(ex).build(); }

Overall exception is: java.lang.NoSuchMethodError: 'org.springframework.boot.actuate.health.Health$Builder org.springframework.boot.actuate.health.Health.down(java.lang.Exception)'