Logstash file input does not deduct the new file after log rotation

Elastic Stack Logstash
1

parser_*.log once this file is 10mb this gets renamed to parser_appserver_backup.log and then creates a new file called parser_appserver.log.later parser_appserver_backup.log is deleted

logstash completely reads parser_appserver_backup.log file. once parser_appserver_backup.log is deleted. It never detects the parser_appserver.log file.

if restarted then the inode entries are updated in sincedb file.

find my config

input {
  file {
    path => "C:/Program Files (x86)/hitachi/app server/DB/Logs/parser_*.log"
    type => "parser"
    start_position => "beginning"
    sincedb_path => "C:/setup/tools/logstash/sincedb_parser"
	sincedb_clean_after => "1w"
	close_older => 60           # Close file descriptors for files not modified in 60 seconds
    ignore_older => 86400
    discover_interval => 15
    stat_interval => 5
    codec => plain { charset => "Windows-1252" }
  }
  file {
    path => "C:/Program Files (x86)/hitachi/app server/DB/Logs/server_*.log"
    type => "server"
    start_position => "beginning"
    sincedb_path => "C:/setup/tools/logstash/sincedb_server"
	sincedb_clean_after => "1w"
    ignore_older => 86400
    discover_interval => 5
    stat_interval => 3
    codec => plain { charset => "Windows-1252" }
  }
}
2

I suggest you let logstash read the log, then shut it down and take a copy of the sincedb. The sincedb file is used to persist the in-memory sincedb, so I would shut logstash down to be absolutely sure that the in-memory and on-disk values match.

Then restart logstash, let the file rotate, shut logstash down, and take another copy of the on-disk sincedb.

Then let the old file get deleted and the new file get rotated again. Then shut logstash down and take another copy of the on-disk sincedb and start examining them to see if you are suffering from inode re-use.

If I use logstash to read the file foo.txt then I get a sincedb entry like

1453373788-430517-10551296 0 0 33 1748277086.392 C:/Path.../foo.txt

"1453373788-430517-10551296" is the unique id used to track the file. This does not change when a file is renamed. The first part (1453373788) is the volume identifier. The other two parts are the high and low parts of the fileid.

At each stage of the above process you can check the fileid of the log file and rotated log file in a shell

c:\Path...>fsutil file queryfileid foo.txt
File ID is 0x000000000000000000a10000000691b5

(you will need admin access if you do not own the file). Note that

0x000691b5 == 430517
0x00a10000 == 10551296

So go through the sincedb files, see if the paths match the fileids, and if the file input is correctly tracking the rotation and possible re-use of the fileid.

The fourth field in the sincedb entry (33 in the example above) is the number of bytes it has read from the fileid. If that is larger than the current size of the file that could be signalling inode re-use.

The first part of the fileid in the sincedb is the volume identifier. If you are mounting the same remote volume and see this change then things are not going to work. For some filesystems the volume id can change each time it is mounted.

1 Like
3

@guru_dev - in your other recent thread on a pretty similar topic, the suggestion was:

your issue is that when the file is rotate it seems that the permissions are being changed, so logstash can not read anymore from the previous file

2 Qs:

a) Have you ruled something similar out here?

b) That other recent thread ended without a clear conclusion, did you resolve that one and if so, how?