Honeytail will not ingest redhat or ubuntu-based syslogs by default.

[jagibson]

Both Redhat and Ubuntu do not include the field at the beginning of their logs by default. However, honeytail will not ingest lines without this field.

When I manually insert the field, honeytail will parse the logs.

As a previous bug report (#142) states, this field is optional and honeytail should ingest the log regardless of whether it's there or not.

Example:

tail msgs.log
Dec 16 15:29:49 ftp17 dbus[1856970]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service'
Dec 16 15:29:49 ftp17 dbus[1856970]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.login1.service': Unit is masked.

honeytail:

/root/go/bin/honeytail --parser=syslog --dataset=test --writekey=****************************** --syslog.mode=rfc3164 --file=msgs.log --backfill --log_level=debug --localtime
DEBU[0000] parsed arguments: main.GlobalOptions.APIHost=https://api.honeycomb.io/,main.GlobalOptions.TailSample=true,main.GlobalOptions.ConfigFile=,main.GlobalOptions.ConfigYaml=,main.GlobalOptions.WriteYaml=,main.GlobalOptions.SampleRate=1,main.GlobalOptions.NumSenders=80,main.GlobalOptions.BatchFrequencyMs=100,main.GlobalOptions.BatchSize=50,main.GlobalOptions.Debug=false,main.GlobalOptions.DebugOut=false,main.GlobalOptions.StatusInterval=60,main.GlobalOptions.Backfill=true,main.GlobalOptions.RebaseTime=false,main.GlobalOptions.Localtime=true,main.GlobalOptions.Timezone=,main.GlobalOptions.ScrubFields=[],main.GlobalOptions.DropFields=[],main.GlobalOptions.AddFields=[],main.GlobalOptions.DAMapFile=,main.GlobalOptions.RequestShape=[],main.GlobalOptions.ShapePrefix=,main.GlobalOptions.RequestPattern=[],main.GlobalOptions.RequestParseQuery=whitelist,main.GlobalOptions.RequestQueryKeys=[],main.GlobalOptions.BackOff=true,main.GlobalOptions.PrefixRegex=,main.GlobalOptions.DeterministicSample=,main.GlobalOptions.DynSample=[],main.GlobalOptions.DynWindowSec=30,main.GlobalOptions.PreSampledField=,main.GlobalOptions.GoalSampleRate=0,main.GlobalOptions.MinSampleRate=1,main.GlobalOptions.JSONFields=[],main.GlobalOptions.FilterFiles=[],main.GlobalOptions.RenameFields=[],main.GlobalOptions.LogLevel=debug,main.RequiredOptions.ParserName=syslog,main.RequiredOptions.WriteKey=[REDACTED],main.RequiredOptions.LogFiles=[msgs.log],main.RequiredOptions.Dataset=test,main.OtherModes.Help=false,main.OtherModes.ListParsers=false,main.OtherModes.Version=false,main.OtherModes.WriteDefaultConfig=false,main.OtherModes.WriteCurrentConfig=false,main.OtherModes.WriteCurrentYaml=false,main.OtherModes.WriteManPage=false,tail.TailOptions.ReadFrom=beginning,tail.TailOptions.Stop=true,tail.TailOptions.Poll=false,tail.TailOptions.StateFile=,tail.TailOptions.HashStateFileDirPaths=false,,csv.Options.Fields=,csv.Options.TimeFieldName=,csv.Options.TimeFieldFormat=,csv.Options.TrimLeadingSpace=false,csv.Options.NumParsers=0,htjson.Options.TimeFieldName=,htjson.Options.TimeFieldFormat=,htjson.Options.NumParsers=0,keyval.Options.TimeFieldName=,keyval.Options.TimeFieldFormat=,keyval.Options.FilterRegex=,keyval.Options.InvertFilter=false,keyval.Options.NumParsers=0,mongodb.Options.LogPartials=false,mongodb.Options.NumParsers=0,mysql.Options.Host=,mysql.Options.User=,mysql.Options.Pass=,mysql.Options.QueryInterval=0,mysql.Options.NumParsers=0,nginx.Options.ConfigFile=,nginx.Options.LogFormatName=,nginx.Options.TimeFieldName=,nginx.Options.TimeFieldFormat=,nginx.Options.NumParsers=0,postgresql.Options.LogLinePrefix=,regex.Options.LineRegex=[],regex.Options.TimeFieldName=,regex.Options.TimeFieldFormat=,regex.Options.NumParsers=0,syslog.Options.Mode=rfc3164,syslog.Options.ProcessList=,syslog.Options.NumParsers=0
INFO[0000] Starting honeytail
INFO[0000] Running in backfill mode may result in rate-limited events for this dataset. This is expected behavior.
Be aware that if you are also sending data from other sources to this dataset, this may result in events
being dropped.
DEBU[0000] about to call tail.TailFile                   conf="{[msgs.log] [] 0 {beginning true false  false}}" location="<nil>" statefile=/tmp/msgs.leash.state tailConf="{<nil> false true false false <nil> false 0 0xc000089230}"
DEBU[0000] attempting to process line                    line="Dec 16 15:29:49 ftp17 dbus[1856970]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service'"
DEBU[0000] attempting to process line                    line="Dec 16 15:29:49 ftp17 dbus[1856970]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.login1.service': Unit is masked."
DEBU[0000] Initializing stats reporting. Will print stats once/60 seconds
DEBU[0000] lines channel is closed, ending syslog processor
INFO[0000] Summary of sent events                        avg_duration=0s count=0 count_per_status="map[]" errors="map[]" fastest=0s lifetime_count=0 response_bodies="map[]" slowest=0s
INFO[0000] Total number of events sent                   number sent by response status code="map[]" total attempted sends=0
INFO[0000] Honeytail is all done, goodbye!

Now I add the field

sed -i -E 's/(^)/<0>\1/g' msgs.log

tail msgs.log
<0>Dec 16 15:29:49 ftp17 dbus[1856970]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service'
<0>Dec 16 15:29:49 ftp17 dbus[1856970]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.login1.service': Unit is masked.

honeytail will now ingest the log:

/root/go/bin/honeytail --parser=syslog --dataset=test --writekey=******************************* --syslog.mode=rfc3164 --file=msgs.log --backfill --log_level=debug --localtime
DEBU[0000] parsed arguments: main.GlobalOptions.APIHost=https://api.honeycomb.io/,main.GlobalOptions.TailSample=true,main.GlobalOptions.ConfigFile=,main.GlobalOptions.ConfigYaml=,main.GlobalOptions.WriteYaml=,main.GlobalOptions.SampleRate=1,main.GlobalOptions.NumSenders=80,main.GlobalOptions.BatchFrequencyMs=100,main.GlobalOptions.BatchSize=50,main.GlobalOptions.Debug=false,main.GlobalOptions.DebugOut=false,main.GlobalOptions.StatusInterval=60,main.GlobalOptions.Backfill=true,main.GlobalOptions.RebaseTime=false,main.GlobalOptions.Localtime=true,main.GlobalOptions.Timezone=,main.GlobalOptions.ScrubFields=[],main.GlobalOptions.DropFields=[],main.GlobalOptions.AddFields=[],main.GlobalOptions.DAMapFile=,main.GlobalOptions.RequestShape=[],main.GlobalOptions.ShapePrefix=,main.GlobalOptions.RequestPattern=[],main.GlobalOptions.RequestParseQuery=whitelist,main.GlobalOptions.RequestQueryKeys=[],main.GlobalOptions.BackOff=true,main.GlobalOptions.PrefixRegex=,main.GlobalOptions.DeterministicSample=,main.GlobalOptions.DynSample=[],main.GlobalOptions.DynWindowSec=30,main.GlobalOptions.PreSampledField=,main.GlobalOptions.GoalSampleRate=0,main.GlobalOptions.MinSampleRate=1,main.GlobalOptions.JSONFields=[],main.GlobalOptions.FilterFiles=[],main.GlobalOptions.RenameFields=[],main.GlobalOptions.LogLevel=debug,main.RequiredOptions.ParserName=syslog,main.RequiredOptions.WriteKey=[REDACTED],main.RequiredOptions.LogFiles=[msgs.log],main.RequiredOptions.Dataset=test,main.OtherModes.Help=false,main.OtherModes.ListParsers=false,main.OtherModes.Version=false,main.OtherModes.WriteDefaultConfig=false,main.OtherModes.WriteCurrentConfig=false,main.OtherModes.WriteCurrentYaml=false,main.OtherModes.WriteManPage=false,tail.TailOptions.ReadFrom=beginning,tail.TailOptions.Stop=true,tail.TailOptions.Poll=false,tail.TailOptions.StateFile=,tail.TailOptions.HashStateFileDirPaths=false,,csv.Options.Fields=,csv.Options.TimeFieldName=,csv.Options.TimeFieldFormat=,csv.Options.TrimLeadingSpace=false,csv.Options.NumParsers=0,htjson.Options.TimeFieldName=,htjson.Options.TimeFieldFormat=,htjson.Options.NumParsers=0,keyval.Options.TimeFieldName=,keyval.Options.TimeFieldFormat=,keyval.Options.FilterRegex=,keyval.Options.InvertFilter=false,keyval.Options.NumParsers=0,mongodb.Options.LogPartials=false,mongodb.Options.NumParsers=0,mysql.Options.Host=,mysql.Options.User=,mysql.Options.Pass=,mysql.Options.QueryInterval=0,mysql.Options.NumParsers=0,nginx.Options.ConfigFile=,nginx.Options.LogFormatName=,nginx.Options.TimeFieldName=,nginx.Options.TimeFieldFormat=,nginx.Options.NumParsers=0,postgresql.Options.LogLinePrefix=,regex.Options.LineRegex=[],regex.Options.TimeFieldName=,regex.Options.TimeFieldFormat=,regex.Options.NumParsers=0,syslog.Options.Mode=rfc3164,syslog.Options.ProcessList=,syslog.Options.NumParsers=0
INFO[0000] Starting honeytail
INFO[0000] Running in backfill mode may result in rate-limited events for this dataset. This is expected behavior.
Be aware that if you are also sending data from other sources to this dataset, this may result in events
being dropped.
DEBU[0000] about to call tail.TailFile                   conf="{[msgs.log] [] 0 {beginning true false  false}}" location="<nil>" statefile=/tmp/msgs.leash.state tailConf="{<nil> false true false false <nil> false 0 0xc00007f230}"
DEBU[0000] attempting to process line                    line="<0>Dec 16 15:29:49 ftp17 dbus[1856970]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service'"
DEBU[0000] Initializing stats reporting. Will print stats once/60 seconds
DEBU[0000] attempting to process line                    line="<0>Dec 16 15:29:49 ftp17 dbus[1856970]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.login1.service': Unit is masked."
DEBU[0000] lines channel is closed, ending syslog processor
DEBU[0000] event send record received                    body= duration=4.40791ms error="<nil>" retry_send=false status_code=202 timestamp="2024-12-16 15:29:49 +0000 UTC"
DEBU[0000] event send record received                    body= duration=4.40791ms error="<nil>" retry_send=false status_code=202 timestamp="2024-12-16 15:29:49 +0000 UTC"
INFO[0000] Summary of sent events                        avg_duration=4.40791ms count=2 count_per_status="map[202:2]" errors="map[]" fastest=4.40791ms lifetime_count=2 response_bodies="map[:2]" slowest=4.40791ms
INFO[0000] Last parsed event                             event="map[content:[system] Activation via systemd failed for unit 'dbus-org.freedesktop.login1.service': Unit is masked. facility:0 hostname:ftp17 priority:0 process:dbus severity:0 timestamp:2024-12-16 15:29:49 +0000 UTC]" event_timestamp="2024-12-16 15:29:49 +0000 UTC"
INFO[0000] Total number of events sent                   number sent by response status code="map[202:2]" total attempted sends=2
INFO[0000] Honeytail is all done, goodbye!

Versions

• Go: go version go1.22.7 linux/amd64
• Honeytail: 1.10.0

Steps to reproduce

1. Try to run honeytail against /var/log/messages on a Redhat or Ubuntu based-system using the options above.

Additional context
Running Amazon Linux 2, had to compile since there was a glibc incompatibility.