Regression - Netty Connection Exceptions with Large Connection Pool Count

[Not-Aru]

Using snapshot: 3.0.0-RC8+22-5ea81ef6-SNAPSHOT

There is a significant regression in request serving when using both ConnectionPoolConfig.Dynamic and ConnectionPoolConfig.Fixed. This seems to be a regression as this issue only appeared after migrating from RC6 to RC8. Exceptions include:

[info] Exception in thread "zio-fiber-823178100,1568885665" io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection reset by peer: /0.0.0.0:8080

[error] Jun 13, 2024 3:25:18 PM io.netty.channel.DefaultChannelPipeline onUnhandledInboundException Jun 13, 2024 3:25:18 PM io.netty.channel.DefaultChannelPipeline onUnhandledInboundException

Exception in thread "zio-fiber-1756387034" io.netty.handler.codec.PrematureChannelClosureException: Channel closed while executing the request. This is likely caused due to a client connection misconfiguration

Reproducer:

package repro

import zio.{ Config => _, *}
import zio.http.*
import zio.http.ZClient.Config
import zio.http.netty.NettyConfig

object Reproducer extends ZIOAppDefault {
  private val clientLayer =
    (ZLayer.succeed(Config.default.copy(connectionPool = ConnectionPoolConfig.Dynamic(1, 512, 60.second))) ++ ZLayer
      .succeed(NettyConfig.defaultWithFastShutdown) ++
      DnsResolver.default) >>> Client.live

  private val url = URL.decode("http://0.0.0.0:8080/Test").toOption.get
  private val req = Request.post(url, Body.empty)

  private val routes =
    HttpApp(
      Routes(
        Method.POST / "Test" ->
          handler(ZIO.sleep(1.second).as(Response.text("success")))
      )
    )

  val run: ZIO[Any & ZIOAppArgs & Scope, Throwable, Unit] = {
    for {
      _ <- Server.serve(routes).fork
      client <- ZIO.service[Client]
      _ <- ZIO.foreachParDiscard(0 to 400) { _ =>
        ZIO
          .scoped[Any] {
            client.request(req).flatMap(_.body.asArray)
          }
          .repeatN(100)
      }
    } yield ()
  }.provide(clientLayer, Server.default)
}

[hearnadam]

This issue first appeared in RC7/RC8. We were not seeing this issue with RC6... @kyri-petrou

[kyri-petrou]

I'll look into it. I think it might have been inadvertently caused by #2860 or #2861

EDIT: Those were merged in RC8, so if RC7 is affected as well that's a different issue

[kyri-petrou]

Regression introduced in #2843, looking into it hopefully I'll have a fix soon

[kyri-petrou]

@Not-Aru @hearnadam I know you're quite interested in the performance of the client, so just FYI re this PR: #2919.

It improves the performance of the client ~45% according to the benchmarks that I added there. Hopefully you'll be able to see that improvement as well in your benchmarks!