Hello,
i want to send data from filebeat (version 9.0.0) to logstash (version 9.0.0) but no matter what I try, it just doesn't work.
info: filebeat and logstash both run on the same virtual machine (Windows Server 2022)
When I start filebeat, I see in the log that it can find the file which should be read, but the data does not arrive at Logstash. Even the output.file does not work.
I tried already the commands "filebeat test config" and "filebeat test output" and both were OK.
The files (.txt) to be read always have the following format and each line should be sent to Logstash as a whole:
25,05,26,07,48,16,01,-9999.999,-9999.999,-9999.999,-9999.999,-9999.999,-9999.999,-9999.999,-9999.999
25,05,26,07,48,18,01,+0008.661,+0008.680,+0005.966,+0005.978,+0008.665,+0008.672,-9999.999,-9999.999
Do you have any idea what the problem could be?
filebeat.yml
#Filebeat inputs
filebeat.inputs:
# filestream is an input for collecting log messages from files.
- type: filestream
# Unique ID among all inputs, an ID is required.
id: my-filestream-id
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- D:/Machineshares/W500_B101E/Spindel/7724/tm-x/result/SD1_013/*.txt
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
# ============================== Filebeat modules ==============================
#filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
#setup.template.settings:
#index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================== Outputs ===================================
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5045"]
#output.file:
#path: "D:/filebeat9.0.0/filebeat-9.0.0-windows-x86_64/logs"
#filename: filebeat
# ================================= Processors =================================
#processors:
# - add_host_metadata:
# when.not.contains.tags: forwarded
# - add_cloud_metadata: ~
# - add_docker_metadata: ~
# - add_kubernetes_metadata: ~
logging.level: debug
logging.to_files: true
logging.files:
path: D:/filebeat9.0.0/filebeat-9.0.0-windows-x86_64/logs
name: filebeatlog
keepfiles: 7
permissions: 0640
filebeat log:
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.335+0200","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure","file.name":"instance/beat.go","file.line":1080},"message":"Home path: [D:\\filebeat9.0.0\\filebeat-9.0.0-windows-x86_64] Config path: [D:\\filebeat9.0.0\\filebeat-9.0.0-windows-x86_64] Data path: [C:\\ProgramData\\filebeat] Logs path: [C:\\ProgramData\\filebeat\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.336+0200","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).loadMeta","file.name":"instance/beat.go","file.line":1204},"message":"Beat metadata path: C:\\ProgramData\\filebeat\\meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.342+0200","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure","file.name":"instance/beat.go","file.line":1088},"message":"Beat ID: e67dc9cb-91bb-4370-9d0f-979a6b1b44cd","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.348+0200","log.logger":"seccomp","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/common/seccomp.loadFilter","file.name":"seccomp/seccomp.go","file.line":97},"message":"Syscall filtering is only supported on Linux","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.348+0200","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater","file.name":"instance/beat.go","file.line":556},"message":"Setup Beat: filebeat; Version: 9.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.348+0200","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).logSystemInfo","file.name":"instance/beat.go","file.line":1641},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"D:\\filebeat9.0.0\\filebeat-9.0.0-windows-x86_64","data":"C:\\ProgramData\\filebeat","home":"D:\\filebeat9.0.0\\filebeat-9.0.0-windows-x86_64","logs":"C:\\ProgramData\\filebeat\\logs"},"type":"filebeat","uuid":"e67dc9cb-91bb-4370-9d0f-979a6b1b44cd"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.348+0200","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).logSystemInfo","file.name":"instance/beat.go","file.line":1650},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"42a721c925857c0d1f4160c977eb5f188e46d425","libbeat":"9.0.0","time":"2025-04-08T14:07:44.000Z","version":"9.0.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.348+0200","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).logSystemInfo","file.name":"instance/beat.go","file.line":1653},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":2,"version":"go1.24.1"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.351+0200","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).logSystemInfo","file.name":"instance/beat.go","file.line":1659},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","native_architecture":"x86_64","boot_time":"2025-05-23T13:35:13+02:00","name":"BUEVM00188","ip":["fe80::1626:5e7c:31c6:70ea","10.6.27.27","::1","127.0.0.1"],"kernel_version":"10.0.20348.3692 (WinBuild.160101.0800)","mac":["00:50:56:9b:7d:e5"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2022 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"20348.3692"},"timezone":"CEST","timezone_offset_sec":7200,"id":"c63dc552-5b55-4af0-92f0-4a67707caa4c"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.351+0200","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).logSystemInfo","file.name":"instance/beat.go","file.line":1688},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"C:\\Windows\\system32","exe":"D:\\filebeat9.0.0\\filebeat-9.0.0-windows-x86_64\\filebeat.exe","name":"filebeat.exe","pid":1520,"ppid":708,"start_time":"2025-05-27T07:48:55.169+0200"},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.354+0200","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater","file.name":"instance/beat.go","file.line":585},"message":"Initializing output plugins","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.359+0200","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).run","file.name":"pipeline/consumer.go","file.line":110},"message":"start pipeline event consumer","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.359+0200","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings","file.name":"pipeline/module.go","file.line":105},"message":"Beat name: BUEVM00188","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.359+0200","log.logger":"modules","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry","file.name":"fileset/modules.go","file.line":136},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-05-27T07:48:55.359+0200","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).setupPipelineLoaderCallback","file.name":"beater/filebeat.go","file.line":196},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.359+0200","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch","file.name":"instance/beat.go","file.line":756},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.359+0200","log.logger":"test","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.isFile","file.name":"registrar/migrate.go","file.line":287},"message":"isFile(C:\\ProgramData\\filebeat\\registry) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.359+0200","log.logger":"test","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.isFile","file.name":"registrar/migrate.go","file.line":287},"message":"isFile() -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.360+0200","log.logger":"test","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.isDir","file.name":"registrar/migrate.go","file.line":280},"message":"isDir(C:\\ProgramData\\filebeat\\registry\\filebeat) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.360+0200","log.logger":"registrar","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.(*Migrator).Run","file.name":"registrar/migrate.go","file.line":82},"message":"Registry type '' found","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.360+0200","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run","file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.360+0200","log.logger":"test","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.isFile","file.name":"registrar/migrate.go","file.line":287},"message":"isFile(.bak) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.362+0200","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop","file.name":"log/log.go","file.line":150},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.362+0200","log.logger":"service","log.origin":{"function":"github.com/elastic/elastic-agent-libs/service.ProcessWindowsControlEvents","file.name":"service/service_windows.go","file.line":134},"message":"Windows is interactive: false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.366+0200","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore","file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\ProgramData\\filebeat\\registry\\filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.371+0200","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore","file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\ProgramData\\filebeat\\registry\\filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.371+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.openStore","file.name":"input-cursor/store.go","file.line":133},"message":"input-cursor::openStore: prefix: azure-blob-storage inputID: ","service.name":"filebeat","input_type":"azure-blob-storage","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.371+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.readStates","file.name":"input-cursor/store.go","file.line":322},"message":"input-cursor store read 0 keys","service.name":"filebeat","input_type":"azure-blob-storage","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.371+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.openStore","file.name":"input-cursor/store.go","file.line":133},"message":"input-cursor::openStore: prefix: httpjson inputID: ","service.name":"filebeat","input_type":"httpjson","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.371+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.readStates","file.name":"input-cursor/store.go","file.line":322},"message":"input-cursor store read 0 keys","service.name":"filebeat","input_type":"httpjson","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.openStore","file.name":"input-cursor/store.go","file.line":133},"message":"input-cursor::openStore: prefix: o365audit inputID: ","service.name":"filebeat","input_type":"o365audit","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.readStates","file.name":"input-cursor/store.go","file.line":322},"message":"input-cursor store read 0 keys","service.name":"filebeat","input_type":"o365audit","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.openStore","file.name":"input-cursor/store.go","file.line":133},"message":"input-cursor::openStore: prefix: salesforce inputID: ","service.name":"filebeat","input_type":"salesforce","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.readStates","file.name":"input-cursor/store.go","file.line":322},"message":"input-cursor store read 0 keys","service.name":"filebeat","input_type":"salesforce","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.openStore","file.name":"input-cursor/store.go","file.line":133},"message":"input-cursor::openStore: prefix: cel inputID: ","service.name":"filebeat","input_type":"cel","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.readStates","file.name":"input-cursor/store.go","file.line":322},"message":"input-cursor store read 0 keys","service.name":"filebeat","input_type":"cel","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.openStore","file.name":"input-cursor/store.go","file.line":133},"message":"input-cursor::openStore: prefix: gcs inputID: ","service.name":"filebeat","input_type":"gcs","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.readStates","file.name":"input-cursor/store.go","file.line":322},"message":"input-cursor store read 0 keys","service.name":"filebeat","input_type":"gcs","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.openStore","file.name":"input-cursor/store.go","file.line":133},"message":"input-cursor::openStore: prefix: winlog inputID: ","service.name":"filebeat","input_type":"winlog","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/input-cursor.readStates","file.name":"input-cursor/store.go","file.line":322},"message":"input-cursor store read 0 keys","service.name":"filebeat","input_type":"winlog","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-05-27T07:48:55.372+0200","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run","file.name":"beater/filebeat.go","file.line":410},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"registrar","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates","file.name":"registrar/registrar.go","file.line":103},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start","file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"registrar","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run","file.name":"registrar/registrar.go","file.line":134},"message":"Starting Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).startInput","file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.path filebeat.inputs.0.paths.0 filebeat.inputs.0.reload.enabled filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"scanner","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.newFileScanner","file.name":"filestream/fswatch.go","file.line":311},"message":"fingerprint mode enabled: offset 0, length 1024","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"scanner","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileScanner).resolveRecursiveGlobs","file.name":"filestream/fswatch.go","file.line":334},"message":"recursive glob enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.372+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.newProspector","file.name":"filestream/prospector_creator.go","file.line":57},"message":"file identity is set to fingerprint","service.name":"filebeat","filestream_id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.375+0200","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).startInput","file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 9208106169933543336)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.375+0200","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start","file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.375+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1","file.name":"compat/compat.go","file.line":135},"message":"Input 'filestream' starting","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:48:55.375+0200","log.logger":"metric_registry","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry","file.name":"inputmon/input.go","file.line":63},"message":"registering","service.name":"filebeat","input_type":"filestream","id":"my-filestream-id","key":"my-filestream-id","uuid":"92f2900e-3b80-4e31-aea8-7c9d85c71d1f","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.375+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileProspector).Run","file.name":"filestream/prospector.go","file.line":208},"message":"Starting prospector","service.name":"filebeat","id":"my-filestream-id","prospector":"file_prospector","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.375+0200","log.logger":"file_watcher","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileWatcher).watch","file.name":"filestream/fswatch.go","file.line":125},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.375+0200","log.logger":"file_watcher","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileWatcher).watch","file.name":"filestream/fswatch.go","file.line":229},"message":"File scan complete","service.name":"filebeat","total":1,"written":0,"truncated":0,"renamed":0,"removed":0,"created":1,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.376+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileProspector).onFSEvent","file.name":"filestream/prospector.go","file.line":254},"message":"A new file D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt has been found","service.name":"filebeat","id":"my-filestream-id","prospector":"file_prospector","operation":"create","source_name":"fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","fingerprint":"12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","os_id":"131072-15738-3725417392","new_path":"D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.376+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*defaultHarvesterGroup).Start","file.name":"input-logfile/harvester.go","file.line":140},"message":"Starting harvester for file","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.377+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).open","file.name":"filestream/input.go","file.line":207},"message":"newLogFileReader with config.MaxBytes:10485760","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","path":"D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt","state-id":"fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:55.378+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*logFile).Read","file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt; Backoff now.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","path":"D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt","state-id":"fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:48:57.379+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*logFile).Read","file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt; Backoff now.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","path":"D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt","state-id":"fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:01.381+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*logFile).Read","file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt; Backoff now.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","path":"D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt","state-id":"fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:05.376+0200","log.logger":"file_watcher","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileWatcher).watch","file.name":"filestream/fswatch.go","file.line":125},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:05.376+0200","log.logger":"file_watcher","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileWatcher).watch","file.name":"filestream/fswatch.go","file.line":229},"message":"File scan complete","service.name":"filebeat","total":1,"written":0,"truncated":0,"renamed":0,"removed":0,"created":0,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:09.381+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*logFile).Read","file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt; Backoff now.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","path":"D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt","state-id":"fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:15.376+0200","log.logger":"file_watcher","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileWatcher).watch","file.name":"filestream/fswatch.go","file.line":125},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:15.377+0200","log.logger":"file_watcher","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileWatcher).watch","file.name":"filestream/fswatch.go","file.line":229},"message":"File scan complete","service.name":"filebeat","total":1,"written":0,"truncated":0,"renamed":0,"removed":0,"created":0,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:19.382+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*logFile).Read","file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt; Backoff now.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","path":"D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt","state-id":"fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-05-27T07:49:25.365+0200","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":156,"time":{"ms":156}},"total":{"ticks":218,"time":{"ms":218},"value":218},"user":{"ticks":62,"time":{"ms":62}}},"info":{"ephemeral_id":"710aca85-e998-4ed5-9ac3-c968a602e111","name":"filebeat","uptime":{"ms":30070},"version":"9.0.0"},"memstats":{"gc_next":42426842,"memory_alloc":27313320,"memory_sys":39340280,"memory_total":41017904,"rss":81895424},"runtime":{"goroutines":33}},"filebeat":{"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash","write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"pipeline":{"clients":1,"events":{"active":0},"queue":{"filled":{"bytes":0,"events":0,"pct":0},"max_bytes":0,"max_events":3200}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"handles":{"open":225}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:25.376+0200","log.logger":"file_watcher","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileWatcher).watch","file.name":"filestream/fswatch.go","file.line":125},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:25.376+0200","log.logger":"file_watcher","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*fileWatcher).watch","file.name":"filestream/fswatch.go","file.line":229},"message":"File scan complete","service.name":"filebeat","total":1,"written":0,"truncated":0,"renamed":0,"removed":0,"created":0,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2025-05-27T07:49:29.383+0200","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*logFile).Read","file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt; Backoff now.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","path":"D:\\Machineshares\\W500_B101E\\Spindel\\7724\\tm-x\\result\\SD1_013\\250526_074816.txt","state-id":"fingerprint::12b8562a8c61d2272382126a0e86985cf086ef6e84bacdeb53c6618ff33e9c7f","ecs.version":"1.6.0"}
For logstash I have the following configuration. I want to write the output in a file (for testing) to see what arrives from filebeat. In the file are just the heartbeat messages, no data from filebeat.
In the logstash log there aren't any errors.
Logstash config:
input {
beats {
port => 5045
}
heartbeat {
id => "cam"
message => "LOGSTASH HEARTBEAT CAM SPINDLE"
interval => 60
add_field => {
topic_id => "prod-w500-logstash-heartbeat-v1"
DataKind => "heartbeat"
}
}
}
output {
file {
path => "${LS_HOME}\\logs\\AllMessages_%{+YYYY-MM-dd}.events.txt"
}
}