Filebeat filestream: Duplicate ingestion after "key with no value" in log.json registry file

Elastic Stack Beats
1

Hi everyone,

I'm seeing a duplicate ingestion issue with Filebeat 8.14 using the filestream input and default settings. For some files, Filebeat suddenly starts reading the whole file again, leading to duplicate events.

While investigating, I noticed something odd in the log.json registry file. Sometimes I see entries like this:

{"k":"filestream::a-b-c::native::528020-161","v":{"ttl":1800000000000,"updated":[516247465673,1747805249],"cursor":{"offset":261852},"meta":null}}                                                                                                                                                                                                                                                                      
{"k":"filestream::a-b-c::native::528020-161"}                                                                                                                                                                                                                                                                                                                                                                               
{"k":"filestream::a-b-c::native::528020-161","v":{"meta":null,"ttl":1800000000000,"updated":[516144192881,1747808794],"cursor":null}}                                                                                                                                                                                                                                                                                      
{"k":"filestream::a-b-c::native::528020-161","v":{"ttl":1800000000000,"updated":[516170057952,1747808794],"cursor":{"offset":262191},"meta":null}}

Here’s what I observed:

  • Sometimes, there appears an entry with just a key and no value, like the second line above.
  • The updated field in the preceding and following lines holds a timestamp. I saw that each of these timestamps matches exactly when a complete re-ingestion happens (the first timestamp for the original ingestion, the second one for the duplicate).
  • Right after the "key with no value", the cursor field (which I believe tracks position in the file) gets reset to null. This seems to cause Filebeat to start from the beginning and re-ingest the whole file.

I checked in Kibana and the timing of these changes matches perfectly with duplicate ingestion spikes :

image
image711×192 6.58 KB

Does this make sense as a possible cause, or should I be looking elsewhere? Has anyone else seen this, or know what could trigger a "key with no value" in the registry?