Logstash output traffic significantly higher than input (minimal transformations) + TCP RST

[vasek (Václav Šulc)]

I’m not very familiar with low-level TCP details, but I would expect Logstash to send a FIN/ACK instead of a RST when closing the connection. This way, the session would close cleanly, and the agent could simply reconnect when it has data to send. Letting the session expire after client_inactivity_timeout makes sense — just not by sending a RST.

I’m glad I wasn’t the only one confused. Thanks a lot for clarifying the situation.

[vasek (Václav Šulc)]

I’ve opened a question specifically about the RST behavior here:

Why does Logstash close idle TCP connections with RST instead of FIN?

Hi Elastic team, I’m trying to better understand how client_inactivity_timeout works in Logstash. I’ve noticed that when this timeout is reached, Logstash closes the TCP connection by sending a RST, even though the connection is still technically alive — TCP keep-alives are being sent by Winlogbeat every ~15 seconds and acknowledged by Logstash (confirmed via Wireshark). As I understand it: client_inactivity_timeout is based only on application-level data (e.g., log events), TCP-level activ…

Hopefully someone from the Elastic team can provide more insight or consider addressing it if it makes sense.

[Badger]

@vasek

Winlogbeat sends a TCP Keep-Alive approximately every 15 seconds, and Logstash responds with a Keep-Alive ACK

I don't think that is quite right. When TCP keepalives are enabled the TCP stack on the end that enabled them will send a keepalive ACK and the receiving TCP stack will ack it (the receiving TCP stack has no idea that it is a keepalive). logstash never sees the keepalive. It's entirely at the TCP level.

Keepalives provide rapid notification of the termination of the remote process or server, and also try to preserve the connection across any stateful components in the network (firewalls, NAT devices, etc.).

The application has no idea that TCP keepalives are happening (except to the extent that it may have enabled them when it opened the connection).

Typically an application would close using RST rather than FIN if it is uninterested in processing any additional data from the connection (including data already received).

[RainTown (Kevin Maguire)]

Thanks for keeping us precise @Badger !!

@vasek

Logstash responds with a Keep-Alive ACK

That wording is not accuraret. There was an ACK, it wasn't logstash's ACK.

And the "ignoring" I mentioned is also a bit dubious wording, logstash didn't see then indeed.

@Badger

The application has no idea that TCP keepalives are happening (except to the extent that it may have enabled them when it opened the connection).

Well yes, it (the application) might have no idea, and it (the TCP layer) just ACKs to packets that contain no data, so dont need processing by the application (logstash in this case).

But I guess both applications could use SO_KEEPALIVE. I'm not informed enough to comment on whether that would be better or not, for any definition of better, but I do know, as @vasek also discovered, that RSTs are a bit untidy when troubleshooting things.