Detailed tasks information as metrics #371

burmanm · 2023-08-31T15:05:39Z

While Cassandra metrics already include compaction and streaming statistics, they do not include per-stream, per-compaction or per-sstabletask metrics which would allow to follow one single compaction's progress for example.

This PR adds those (a lot of extra work is because of 3.11 support, which has different APIs than 4.0 and up that would have simplified this implementation).

Fixes #293

… yet) and modify some methods to be more reusable

adejanovski · 2023-09-29T11:50:40Z

...nt-api-agent-common/src/main/java/io/k8ssandra/metrics/prometheus/CassandraTasksExports.java

+                      && operationType != OperationType.KEY_CACHE_SAVE
+                      && operationType != OperationType.ROW_CACHE_SAVE;
+                })
+            .collect(Collectors.toList());


Issue: I'm getting the following exception when running in Cassandra 4.0:

java.lang.IllegalArgumentException: No enum constant org.apache.cassandra.db.compaction.OperationType.Compaction at java.base/java.lang.Enum.valueOf(Unknown Source) at org.apache.cassandra.db.compaction.OperationType.valueOf(OperationType.java:20) at io.k8ssandra.metrics.prometheus.CassandraTasksExports.lambda$getCompactionStats$0(CassandraTasksExports.java:288) at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source) at io.k8ssandra.metrics.prometheus.CassandraTasksExports.getCompactionStats(CassandraTasksExports.java:293) at io.k8ssandra.metrics.prometheus.CassandraTasksExports.collect(CassandraTasksExports.java:46) at io.prometheus.client.Collector.collect(Collector.java:45) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:204) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:219) at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:152) at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:71) at io.prometheus.client.exporter.common.TextFormat.writeFormat(TextFormat.java:53) at io.k8ssandra.metrics.http.NettyServerHandler.channelRead0(NettyServerHandler.java:66) at io.k8ssandra.metrics.http.NettyServerHandler.channelRead0(NettyServerHandler.java:30) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source)

Okay, I added .toUpperCase() and a catch clause to prevent this Exception. Should help with 4.0 as the CompactionMetrics calls seem to work reliably only with 4.1 and newer.

adejanovski · 2023-09-29T11:52:10Z

...nt-api-agent-common/src/main/java/io/k8ssandra/metrics/prometheus/CassandraTasksExports.java

+    familySamples.addAll(getStreamInfoStats());
+
+    // Collect other sstableOperations (if not part of Compactions metrics already)
+


Todo: we need follow up tickets for these missing types of operations

At least on 4.1 we have everything, for example UpgradeSSTables is here:

https://github.com/apache/cassandra/blob/8cc2b481922dd77a9956e76aca1220aeb799953b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L610

adejanovski

Issue: I'm getting exceptions when hitting the metrics endpoint if a compaction is running:

WARN  [epollEventLoopGroup-6-1] 2023-10-04 08:56:11,917 DefaultChannelPipeline.java:1152 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.IndexOutOfBoundsException: Index 5 out of bounds for length 5
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Unknown Source)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Unknown Source)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Unknown Source)
	at java.base/java.util.Objects.checkIndex(Unknown Source)
	at java.base/java.util.ArrayList.get(Unknown Source)
	at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:119)
	at io.prometheus.client.exporter.common.TextFormat.writeFormat(TextFormat.java:53)
	at io.k8ssandra.metrics.http.NettyServerHandler.channelRead0(NettyServerHandler.java:66)
	at io.k8ssandra.metrics.http.NettyServerHandler.channelRead0(NettyServerHandler.java:30)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)

A pretty similar exception happens when a streaming operation is ongoing:

WARN  [epollEventLoopGroup-6-1] 2023-10-04 09:23:05,143 DefaultChannelPipeline.java:1152 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.lang.IndexOutOfBoundsException: Index 4 out of bounds for length 4
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Unknown Source)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Unknown Source)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Unknown Source)
	at java.base/java.util.Objects.checkIndex(Unknown Source)
	at java.base/java.util.ArrayList.get(Unknown Source)
	at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:119)
	at io.prometheus.client.exporter.common.TextFormat.writeFormat(TextFormat.java:53)
	at io.k8ssandra.metrics.http.NettyServerHandler.channelRead0(NettyServerHandler.java:66)
	at io.k8ssandra.metrics.http.NettyServerHandler.channelRead0(NettyServerHandler.java:30)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)

…s to verify the methods do not error and have correct labelValues and labelNames sizes in the end results also.

adejanovski

It's working nicely now

burmanm added 4 commits August 30, 2023 18:36

Add compaction stats, add metrics to the JobExecutor (but not exposed…

e023c1a

… yet) and modify some methods to be more reusable

Add parameter to allow disabling gather of extended metrics

781b5fa

Add StreamInfo metrics

7da5c26

Fix NPE

192f020

burmanm marked this pull request as ready for review September 22, 2023 09:22

adejanovski self-requested a review September 29, 2023 11:31

adejanovski requested changes Sep 29, 2023

View reviewed changes

burmanm added 2 commits October 3, 2023 14:29

Improve backwards compatibility

6f94a4b

Fix formatting

bcd0014

adejanovski requested changes Oct 4, 2023

View reviewed changes

burmanm added 4 commits October 4, 2023 16:34

Fix labelValues adder (defaultLabelValuse were overwritten), add test…

c558045

…s to verify the methods do not error and have correct labelValues and labelNames sizes in the end results also.

Add missing license header

9d689d0

Fix naming of _total metric

c06a117

Add more defensives

c4f4b22

adejanovski approved these changes Oct 9, 2023

View reviewed changes

burmanm merged commit a80428f into master Oct 9, 2023
47 checks passed

burmanm deleted the tasks_to_metrics branch October 9, 2023 16:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detailed tasks information as metrics #371

Detailed tasks information as metrics #371

burmanm commented Aug 31, 2023

adejanovski Sep 29, 2023

burmanm Oct 3, 2023

adejanovski Sep 29, 2023

burmanm Oct 3, 2023

adejanovski left a comment

adejanovski left a comment

		familySamples.addAll(getStreamInfoStats());

		// Collect other sstableOperations (if not part of Compactions metrics already)

Detailed tasks information as metrics #371

Detailed tasks information as metrics #371

Conversation

burmanm commented Aug 31, 2023

adejanovski Sep 29, 2023

Choose a reason for hiding this comment

burmanm Oct 3, 2023

Choose a reason for hiding this comment

adejanovski Sep 29, 2023

Choose a reason for hiding this comment

burmanm Oct 3, 2023

Choose a reason for hiding this comment

adejanovski left a comment

Choose a reason for hiding this comment

adejanovski left a comment

Choose a reason for hiding this comment