Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AbstractTcpTransport connection reset by peer when using PanOS 9 input as a source #954

Open
ed-graylog opened this issue Dec 17, 2021 · 5 comments

Comments

@ed-graylog
Copy link

Expected Behavior

As a Graylog user when I create a PanOS 8 or 9 input, I expect it to work out of the box without any issues.

Current Behavior

Currently when looking through the Graylog master logs we see the following message over and over
(cause java.io.IOException: Connection reset by peer)

Possible Solution

I'm unclear exactly as to why this is occurring. We know the customer is behind an F5 load balancer however we have attempted using keep alive and not keep alive settings on the inputs. We have additionally attempted using both PanUS 8 and 9 and still got the same errors. Strangely when using raw tcp input we did not get this error however the fields were not created and the time was incorrect when using raw input. We also checked the logs for Graylog and also for F5 and found nothing of note.

Steps to Reproduce (for bugs)

  1. Please see HS-635292232 for additional notes and context.
@user29835461
Copy link

From my own research I'd say most likely this is due Netty logging when for performance optimizations some network device or server doesn't fully close the TCP connections. It is harmless, although it is irritating to see error spam in the logs.

It is possible to configure log4j logging level for Netty to FATAL, although this might eat also other ERRORs.

@bernd bernd transferred this issue from Graylog2/graylog2-server Dec 20, 2021
@bernd
Copy link
Member

bernd commented Dec 20, 2021

@edward-marrufo Thank you! I moved this over to the integrations repository where the code lives.

@waab76
Copy link
Contributor

waab76 commented Dec 20, 2021

This appears to be an issue with how we are handling the logging for these connection reset events. In our Netty exception logging handler, we would attempt to log these events at the trace level if trace logging is enabled. However, since trace logging is presumably not enabled in this case, the events are getting logged at the error level: https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog2/inputs/transports/netty/ExceptionLoggingChannelHandler.java#L42-L47

I think we should take another look at the exception logging handler and consider logging these events at trace if trace is enabled or not logging them at all if trace is disabled.

@ed-graylog
Copy link
Author

From a support perspective, does this mean there is no legitimate issue, it is merely a harmless logging issue at this point?

@dio99
Copy link

dio99 commented May 12, 2022

same is if the raw syslog input too.

2022-05-12 13:13:36,801 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Raw/Plaintext TCP/618241d06c7ad24c5315a989] (channel [id: 0xd360c1af, L:/172.16.2.116:514 ! R:/172.16.2.14:32894]) (cause io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer)
2022-05-12 13:13:39,786 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Raw/Plaintext TCP/618241d06c7ad24c5315a989] (channel [id: 0x864ddd0f, L:/172.16.2.116:514 ! R:/172.16.2.14:32946]) (cause io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants