Delayed shutdown of filebeat only after finished processing log files

Elastic Stack Beats
1

I've not been able to find any reference to this in the docs, maybe it doesn't exist, or maybe my searchfu is failing me: is there a way to instruct filebeat to shutdown only after it's caught up with all the log files it's been told to monitor?

This feels like something that should exist in some form/I can't be the first person to think "it'd be nice if I could shut this down - but only after it's caught up with the work I needed it to do" ...

e.g. in a simple system using supervisord, one process generating logfiles and filebeat processing them, is there a way to make the following sequence of commands result in the filebeat process only terminating after it's shipped all log messages in the files it's watching:

# start app and filebeat
supervisorctl start myapp
supervisorctl start filebeat

# stop app and filebeat
supervisorctl stop myapp
# app is stopped, but not all the logs are necessarily shipped yet
supervisorctl stop filebeat
# app is stopped, all the logs generated were shipped, filebeat is stopped

I don't mind whether it's something that can be passed to filebeat as a configuration setting that causes it to delay it's shutdown after it receives the signal, or whether it's something that could be put into a wrapper shutdown script that would prevent the signal going direct to filebeat - but I can't work out how to do either atm :slight_smile:

2

I think this may work if you manually run filebeat with the --once option as mentioned in the documentation.

But there are some catches, it does not work if the input is the new filestream input, you would need to use the old log input, and this is a deprecated input that will be removed in the future.

For now you can use it because the input still exist, but in future versions it may be removed.