HTTP/2 Benchmarks on reactor netty 2 don't work

[pderop]

Expected Behavior

Be able to run some Gatling HTTP/2 bench using reactor netty 2 and netty 5.

Actual Behavior

HTTP/2 gatling benchmarks are failing with errors like this:

> Request timeout to 127.0.0.1/127.0.0.1:8080 after 60000 ms      15535 (62.97%)
> i.n.c.ConnectTimeoutException: connection timed out: /127.0.0.   8816 (35.74%)
1:8080
> i.n.h.s.SslHandshakeTimeoutException: handshake timed out after    319 ( 1.29%)
r 10000ms

Steps to Reproduce

• clone https://github.com/reactor/benchmarks
• Install JDK19
• build (./gradlew build)
• on one console, start the reactor netty 2 frontend using HTTP/2 protocol:

java -DPROTOCOL=H2 -jar frontend-rn-20x/build/libs/frontend-rn-20x-1.0.0.jar

• on another console, run Gatling:

rm -rf test-reports/; java -DPROTOCOL=H2 -DURATION=10 -jar gatling/build/libs/gatling-1.0.0-all.jar application “test” TextPlain

After few minutes, some errors will be displayed like "handshake timed out".

While the test is running, jstat seems to indicate that the frontend is having some heap memory leak:

    MC         MU       CCSC      CCSU         OC           OU       YGC    FGC    FGCT     CGC    CGCT       GCT   
   24704.0    24333.0    2944.0    2781.2     155648.0     106984.5     10     0     0.000     6     0.007     0.028
   24896.0    24493.3    2944.0    2781.7     172032.0     107161.0     21     0     0.000     6     0.007     0.051
   24960.0    24632.7    2944.0    2783.1     507904.0     197459.5     34     0     0.000     6     0.007     0.160
   24960.0    24636.7    2944.0    2784.0    1638400.0     754936.5     40     0     0.000     6     0.007     0.363
   24960.0    24638.4    2944.0    2784.0    2744320.0    1550304.0     45     0     0.000     6     0.007     0.631
   24960.0    24638.4    2944.0    2784.0    2670592.0    1759994.5     50     0     0.000     6     0.007     0.774
   24960.0    24638.4    2944.0    2784.0    3190784.0    2045706.0     54     0     0.000     6     0.007     0.946
   24960.0    24638.7    2944.0    2784.0    3739648.0    1920992.0     58     0     0.000     8     0.008     1.247
   25088.0    24686.6    2944.0    2784.0    3833856.0    2117195.5     62     0     0.000     8     0.008     1.473
   25088.0    24742.3    2944.0    2784.5    4947968.0    3147744.0     67     0     0.000     9     0.010     2.010
   25152.0    24803.0    2944.0    2784.5    6152192.0    4818912.0     71     0     0.000     9     0.010     2.416
   25152.0    24805.2    2944.0    2784.5    7303168.0    6281184.0     81     0     0.000    10     0.010     2.999
   25152.0    24805.2    2944.0    2784.5    7565312.0    6135776.0     90     0     0.000    11     0.013     3.440
   25152.0    24805.2    2944.0    2784.5    7393280.0    5818336.0     99     0     0.000    12     0.013     4.085
   25152.0    24805.2    2944.0    2784.5    7049216.0    5103584.0    110     0     0.000    13     0.015     4.677
   25280.0    24872.6    2944.0    2784.5    7946240.0    6973408.0    116     0     0.000    13     0.015     5.307
   25280.0    24879.1    2944.0    2784.5    8318976.0    8315002.5    135     0     0.000    13     0.015     6.234
   25280.0    24778.9    2944.0    2755.4    5996544.0    3319548.6    139     1     0.450    13     0.015     6.900
   25280.0    24838.7    2944.0    2755.4    7487488.0    5935374.1    143     1     0.450    13     0.015     7.540
   25280.0    24838.7    2944.0    2755.4    8355840.0    8354581.7    196     2     0.450    13     0.015     8.299
   25280.0    24838.7    2944.0    2755.4    7073792.0    5211586.1    198     2     1.121    13     0.015     9.261
   25280.0    24838.7    2944.0    2755.4    7946240.0    7765442.1    208     2     1.121    13     0.015     9.967
   25280.0    24849.1    2944.0    2755.4    7348224.0    5662974.5    251     3     1.970    13     0.015    11.251
   25344.0    24916.1    2944.0    2755.4    8237056.0    8235258.5    268     3     1.970    13     0.015    11.987
   25344.0    24916.1    2944.0    2755.4    7946240.0    6956097.3    290     4     2.927    13     0.015    13.330
   25344.0    24920.2    2944.0    2755.4    8364032.0    8363073.3    350     5     2.927    13     0.015    13.902
   25344.0    24966.4    2944.0    2755.4    8359936.0    8358496.2    412     6     3.999    13     0.015    15.565
   25344.0    24967.5    2944.0    2755.4    8364032.0    8363298.6    456     7     5.088    13     0.015    17.063
   25344.0    24967.5    2944.0    2755.4    8310784.0    8309361.3    458     8     6.204    13     0.015    18.271
   25344.0    24969.1    2944.0    2755.4    8318976.0    8314907.8    462    10     8.350    13     0.015    20.592
   25344.0    25008.7    2944.0    2760.2    8318976.0    8315733.3    464    11     9.458    13     0.015    21.802

Using the following Netty5 debug options don't report anything:

-Dio.netty5.leakDetection.level=paranoid 
-Dio.netty5.leakDetection.targetRecords=32 
-Dio.netty5.buffer.lifecycleTracingEnabled=true 
-Dio.netty5.buffer.leakDetectionEnabled=true 

Possible Solution

Your Environment

Java 19
Benchmarks project
Reactor Netty 2
Netty 5

[pderop]

Here is a screenshot of memory analyzer on the frontend.

[pderop]

some updates: the frontend test server was not setting any max concurrent streams, and specify a max-concurrent-stream on the frontend server avoids the memory leak.

Now, the benchmark still fails with this exception that happens on the frontend server (using the "TextPlain" scenario:

16:35:04.683 [reactor-http-nio-10] WARN  i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty5.handler.codec.http2.Http2Exception: Stream 13973 does not exist
	at io.netty5.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:99)
	at io.netty5.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.verifyStreamMayHaveExisted(DefaultHttp2ConnectionDecoder.java:667)
	at io.netty5.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:427)
	at io.netty5.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:513)
	at io.netty5.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:256)
	at io.netty5.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
	at io.netty5.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174)
	at io.netty5.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61)
	at io.netty5.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
	at io.netty5.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
	at io.netty5.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:387)
	at io.netty5.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:330)
	at io.netty5.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:204)
	at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
	at io.netty5.channel.DefaultChannelHandlerContext.findAndInvokeChannelRead(DefaultChannelHandlerContext.java:445)
	at io.netty5.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:426)
	at reactor.netty5.http.server.HttpServerConfig$H2ChannelMetricsHandler.channelRead(HttpServerConfig.java:717)
	at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
	at io.netty5.channel.DefaultChannelHandlerContext.findAndInvokeChannelRead(DefaultChannelHandlerContext.java:445)
	at io.netty5.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:426)
	at io.netty5.channel.internal.DelegatingChannelHandlerContext.fireChannelRead(DelegatingChannelHandlerContext.java:113)
	at io.netty5.handler.codec.ByteToMessageDecoder$ByteToMessageDecoderContext.fireChannelRead(ByteToMessageDecoder.java:446)
	at io.netty5.handler.ssl.SslHandler.unwrap(SslHandler.java:1215)
	at io.netty5.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1081)
	at io.netty5.handler.ssl.SslHandler.decode(SslHandler.java:1128)
	at io.netty5.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:387)
	at io.netty5.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:330)
	at io.netty5.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:204)
	at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
	at io.netty5.channel.DefaultChannelHandlerContext.findAndInvokeChannelRead(DefaultChannelHandlerContext.java:445)
	at io.netty5.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:426)
	at io.netty5.channel.ChannelHandler.channelRead(ChannelHandler.java:235)
	at io.netty5.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:455)
	at io.netty5.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:838)
	at io.netty5.channel.AbstractChannel$ReadSink.processRead(AbstractChannel.java:1996)
	at io.netty5.channel.nio.AbstractNioByteChannel.doReadNow(AbstractNioByteChannel.java:74)
	at io.netty5.channel.AbstractChannel$ReadSink.readLoop(AbstractChannel.java:2061)
	at io.netty5.channel.AbstractChannel.readNow(AbstractChannel.java:910)
	at io.netty5.channel.nio.AbstractNioChannel.access$100(AbstractNioChannel.java:42)
	at io.netty5.channel.nio.AbstractNioChannel$1.handle(AbstractNioChannel.java:107)
	at io.netty5.channel.nio.NioHandler.processSelectedKey(NioHandler.java:506)
	at io.netty5.channel.nio.NioHandler.processSelectedKeysOptimized(NioHandler.java:489)
	at io.netty5.channel.nio.NioHandler.processSelectedKeys(NioHandler.java:430)
	at io.netty5.channel.nio.NioHandler.run(NioHandler.java:407)
	at io.netty5.channel.SingleThreadEventLoop.runIO(SingleThreadEventLoop.java:192)
	at io.netty5.channel.SingleThreadEventLoop.run(SingleThreadEventLoop.java:176)
	at io.netty5.util.concurrent.SingleThreadEventExecutor.lambda$doStartThread$4(SingleThreadEventExecutor.java:773)
	at io.netty5.util.internal.ThreadExecutorMap.lambda$apply$1(ThreadExecutorMap.java:68)
	at io.netty5.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1589)