Http Client threads remain open

[pschmid]

Hi,
After updating apicurio client from apicurio-registry-utils-serde:1.3.2.Final to apicurio-registry-serdes-avro-serde:2.1.4.Final I'm seeing a big number of threads that remain open that I was not seeing in the previous version.
The mentioned threads are SelectorManager threads as shown below.

"HttpClient-100-SelectorManager" #583 daemon prio=5 os_prio=31 cpu=3.00ms elapsed=47.59s tid=0x00007fa5f26ae000 nid=0x57e07 runnable [0x000070001c2a3000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueue.poll([email protected]/Native Method)
at sun.nio.ch.KQueueSelectorImpl.doSelect([email protected]/KQueueSelectorImpl.java:122)
at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:124)
- locked <0x0000000710901828> (a sun.nio.ch.Util$2)
- locked <0x00000007109017c8> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:136)
at jdk.internal.net.http.HttpClientImpl$SelectorManager.run([email protected]/HttpClientImpl.java:867)

Is this something expected or there is some http client connections that is not being closed correctly?
Regards,
Pablo

[EricWittmann]

Yep - we introduced a problem in the HTTP client. The good news is we fixed it and everything should be good in 2.1.5.Final. Can you give that a try and report back? That would be super helpful.

[pschmid]

Thanks for the answer. I will need to run tests again to check how many http clients are being instantiated. In the meantime, I would like to understand what the SelectorManager is. From my previous run, the new threads that I'm seeing in 2.1.5.Final are related to HttpClient-100-SelectorManager and in apicurio-registry-utils-serde:1.3.2.Final I was not seeing any thread related to SelectorManager

[carlesarnal]

You're not gonna see them because the rest client used in 1.3.2.Final was based on Retrofit. Starting with 2.x, the rest client is based on the java 11 httclient. If you're curious about the SelectorManager, is "just" a thread class with extended functionality about the rest client. As a summary, we moved from an external library to the http client brought by the platform.

[tomasAlabes]

Hi Carles, what's happening to use is that we have a good amount of producers/consumers that are active, and since 2.x we've seen our threads count double (because spring-kafka also maintains 1 thread per producer/consumer).
Is it necessary to have N apicurio clients? Being that it is a producer/consumer agnostic REST call to Apicurio, to read/register schemas (we use content id's btw). I don't seem to see the benefit, at least in our case. Could it be configurable at least?

[carlesarnal]

Hi Tomas. As you can imagine from my comments, we started a refactor of the registry client with 2.x. With this refactor, as I explained above, we replaced the http client from retrofit by the java one. Now direct to your question, is for sure in our backlog to build some configuration options around the registry client like e.g. providing your own ExecutorService, configuring the maximum number of connections etc. That said, the number of configuration options in our serdes layer is already significant, so we want to be careful about the number of options related to the rest client that we want to expose through the serdes. That said, if you're missing something specific, feel free to open an issue and we can discuss the idea there.

[tomasAlabes]

Thanks for the answer Carles. I understand, my concern is that going from 1 on-demand rest call to 1 thread per consumer/producer is a huge change on services consuming many topics/partitions. To give you an example, our service had ~1500 threads consuming/producing, and now it got doubled and our stress tests started failing with the cpu spiking with ~3k threads.
My main question is, being that is a plain REST call, do we need such a thread pool to do them? I can start an issue with this to formalize the question.