We have a setup where a Logstash machine (v7.17.28) sends to a Splunk Heavy Forwarder using HTTP.
We see a lot of HTTP Output errors and I am unable to find the cause. All data seems to arrive correctly.
The error is (I've anonimised the url's):
[2025-06-03T06:47:37,973][ERROR][logstash.outputs.http ][mypipeline] [HTTP Output Failure] Could not fetch URL {:url=>"mysplunkurl:8088/services/collector/event/1.0", :method=>:post, :message=>"mysplunkurl:8088 failed to respond", :class=>"Manticore::ClientProtocolException", :will_retry=>true}
I have investigated the TCP traffic going between the two servers and that contains virtually no errors. At any point, there are between 85-110 concurrent connections to the Heavy Forwarder from Logstash. We use about 15 pipelines that use the HTTP output module. There are no persistent queues, system load is low with 16 pipeline workers.
I have tried increasing the batch size but that showed no obvious effect.
What next step could I take either to find the cause or to optimize things to get rid of this problem?
Thanks in advance.