syslog ingestion ignores timezone, localtime flags

[jagibson]

I'm trying to ingest logs that are in rfc3164 format with honeytail. The timezone of our logs is CST. However honeytail always ingest them as UTC.

Example log

tail msgs.log
<0>Dec 16 15:14:12 ftp17 dbus[1856970]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service'
<0>Dec 16 15:14:12 ftp17 dbus[1856970]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.login1.service': Unit is masked.

date
Mon Dec 16 15:14:49 CST 2024

Using timezone=

/root/go/bin/honeytail --parser=syslog --dataset=test --writekey=******************************* --syslog.mode=rfc3164 --file=msgs.log --backfill --log_level=debug --timezone='America/Chicago'
DEBU[0000] parsed arguments: main.GlobalOptions.APIHost=https://api.honeycomb.io/,main.GlobalOptions.TailSample=true,main.GlobalOptions.ConfigFile=,main.GlobalOptions.ConfigYaml=,main.GlobalOptions.WriteYaml=,main.GlobalOptions.SampleRate=1,main.GlobalOptions.NumSenders=80,main.GlobalOptions.BatchFrequencyMs=100,main.GlobalOptions.BatchSize=50,main.GlobalOptions.Debug=false,main.GlobalOptions.DebugOut=false,main.GlobalOptions.StatusInterval=60,main.GlobalOptions.Backfill=true,main.GlobalOptions.RebaseTime=false,main.GlobalOptions.Localtime=false,main.GlobalOptions.Timezone=America/Chicago,main.GlobalOptions.ScrubFields=[],main.GlobalOptions.DropFields=[],main.GlobalOptions.AddFields=[],main.GlobalOptions.DAMapFile=,main.GlobalOptions.RequestShape=[],main.GlobalOptions.ShapePrefix=,main.GlobalOptions.RequestPattern=[],main.GlobalOptions.RequestParseQuery=whitelist,main.GlobalOptions.RequestQueryKeys=[],main.GlobalOptions.BackOff=true,main.GlobalOptions.PrefixRegex=,main.GlobalOptions.DeterministicSample=,main.GlobalOptions.DynSample=[],main.GlobalOptions.DynWindowSec=30,main.GlobalOptions.PreSampledField=,main.GlobalOptions.GoalSampleRate=0,main.GlobalOptions.MinSampleRate=1,main.GlobalOptions.JSONFields=[],main.GlobalOptions.FilterFiles=[],main.GlobalOptions.RenameFields=[],main.GlobalOptions.LogLevel=debug,main.RequiredOptions.ParserName=syslog,main.RequiredOptions.WriteKey=[REDACTED],main.RequiredOptions.LogFiles=[msgs.log],main.RequiredOptions.Dataset=test,main.OtherModes.Help=false,main.OtherModes.ListParsers=false,main.OtherModes.Version=false,main.OtherModes.WriteDefaultConfig=false,main.OtherModes.WriteCurrentConfig=false,main.OtherModes.WriteCurrentYaml=false,main.OtherModes.WriteManPage=false,tail.TailOptions.ReadFrom=beginning,tail.TailOptions.Stop=true,tail.TailOptions.Poll=false,tail.TailOptions.StateFile=,tail.TailOptions.HashStateFileDirPaths=false,,csv.Options.Fields=,csv.Options.TimeFieldName=,csv.Options.TimeFieldFormat=,csv.Options.TrimLeadingSpace=false,csv.Options.NumParsers=0,htjson.Options.TimeFieldName=,htjson.Options.TimeFieldFormat=,htjson.Options.NumParsers=0,keyval.Options.TimeFieldName=,keyval.Options.TimeFieldFormat=,keyval.Options.FilterRegex=,keyval.Options.InvertFilter=false,keyval.Options.NumParsers=0,mongodb.Options.LogPartials=false,mongodb.Options.NumParsers=0,mysql.Options.Host=,mysql.Options.User=,mysql.Options.Pass=,mysql.Options.QueryInterval=0,mysql.Options.NumParsers=0,nginx.Options.ConfigFile=,nginx.Options.LogFormatName=,nginx.Options.TimeFieldName=,nginx.Options.TimeFieldFormat=,nginx.Options.NumParsers=0,postgresql.Options.LogLinePrefix=,regex.Options.LineRegex=[],regex.Options.TimeFieldName=,regex.Options.TimeFieldFormat=,regex.Options.NumParsers=0,syslog.Options.Mode=rfc3164,syslog.Options.ProcessList=,syslog.Options.NumParsers=0
INFO[0000] Starting honeytail
INFO[0000] Running in backfill mode may result in rate-limited events for this dataset. This is expected behavior.
Be aware that if you are also sending data from other sources to this dataset, this may result in events
being dropped.
DEBU[0000] about to call tail.TailFile                   conf="{[msgs.log] [] 0 {beginning true false  false}}" location="<nil>" statefile=/tmp/msgs.leash.state tailConf="{<nil> false true false false <nil> false 0 0xc00010d230}"
DEBU[0000] attempting to process line                    line="<0>Dec 16 15:14:12 ftp17 dbus[1856970]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service'"
DEBU[0000] Initializing stats reporting. Will print stats once/60 seconds
DEBU[0000] attempting to process line                    line="<0>Dec 16 15:14:12 ftp17 dbus[1856970]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.login1.service': Unit is masked."
DEBU[0000] lines channel is closed, ending syslog processor
DEBU[0000] event send record received                    body= duration=55.573139ms error="<nil>" retry_send=false status_code=202 timestamp="2024-12-16 15:14:12 +0000 UTC"
DEBU[0000] event send record received                    body= duration=815.203118ms error="<nil>" retry_send=false status_code=202 timestamp="2024-12-16 15:14:12 +0000 UTC"
INFO[0000] Summary of sent events                        avg_duration=435.388128ms count=2 count_per_status="map[202:2]" errors="map[]" fastest=55.573139ms lifetime_count=2 response_bodies="map[:2]" slowest=815.203118ms
INFO[0000] Last parsed event                             event="map[content:[system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service' facility:0 hostname:ftp17 priority:0 process:dbus severity:0 timestamp:2024-12-16 15:14:12 +0000 UTC]" event_timestamp="2024-12-16 15:14:12 +0000 UTC"
INFO[0000] Total number of events sent                   number sent by response status code="map[202:2]" total attempted sends=2

Using localtime

/root/go/bin/honeytail --parser=syslog --dataset=test --writekey=************************************** --syslog.mode=rfc3164 --file=msgs.log --backfill --log_level=debug --localtime
DEBU[0000] parsed arguments: main.GlobalOptions.APIHost=https://api.honeycomb.io/,main.GlobalOptions.TailSample=true,main.GlobalOptions.ConfigFile=,main.GlobalOptions.ConfigYaml=,main.GlobalOptions.WriteYaml=,main.GlobalOptions.SampleRate=1,main.GlobalOptions.NumSenders=80,main.GlobalOptions.BatchFrequencyMs=100,main.GlobalOptions.BatchSize=50,main.GlobalOptions.Debug=false,main.GlobalOptions.DebugOut=false,main.GlobalOptions.StatusInterval=60,main.GlobalOptions.Backfill=true,main.GlobalOptions.RebaseTime=false,main.GlobalOptions.Localtime=true,main.GlobalOptions.Timezone=,main.GlobalOptions.ScrubFields=[],main.GlobalOptions.DropFields=[],main.GlobalOptions.AddFields=[],main.GlobalOptions.DAMapFile=,main.GlobalOptions.RequestShape=[],main.GlobalOptions.ShapePrefix=,main.GlobalOptions.RequestPattern=[],main.GlobalOptions.RequestParseQuery=whitelist,main.GlobalOptions.RequestQueryKeys=[],main.GlobalOptions.BackOff=true,main.GlobalOptions.PrefixRegex=,main.GlobalOptions.DeterministicSample=,main.GlobalOptions.DynSample=[],main.GlobalOptions.DynWindowSec=30,main.GlobalOptions.PreSampledField=,main.GlobalOptions.GoalSampleRate=0,main.GlobalOptions.MinSampleRate=1,main.GlobalOptions.JSONFields=[],main.GlobalOptions.FilterFiles=[],main.GlobalOptions.RenameFields=[],main.GlobalOptions.LogLevel=debug,main.RequiredOptions.ParserName=syslog,main.RequiredOptions.WriteKey=[REDACTED],main.RequiredOptions.LogFiles=[msgs.log],main.RequiredOptions.Dataset=test,main.OtherModes.Help=false,main.OtherModes.ListParsers=false,main.OtherModes.Version=false,main.OtherModes.WriteDefaultConfig=false,main.OtherModes.WriteCurrentConfig=false,main.OtherModes.WriteCurrentYaml=false,main.OtherModes.WriteManPage=false,tail.TailOptions.ReadFrom=beginning,tail.TailOptions.Stop=true,tail.TailOptions.Poll=false,tail.TailOptions.StateFile=,tail.TailOptions.HashStateFileDirPaths=false,,csv.Options.Fields=,csv.Options.TimeFieldName=,csv.Options.TimeFieldFormat=,csv.Options.TrimLeadingSpace=false,csv.Options.NumParsers=0,htjson.Options.TimeFieldName=,htjson.Options.TimeFieldFormat=,htjson.Options.NumParsers=0,keyval.Options.TimeFieldName=,keyval.Options.TimeFieldFormat=,keyval.Options.FilterRegex=,keyval.Options.InvertFilter=false,keyval.Options.NumParsers=0,mongodb.Options.LogPartials=false,mongodb.Options.NumParsers=0,mysql.Options.Host=,mysql.Options.User=,mysql.Options.Pass=,mysql.Options.QueryInterval=0,mysql.Options.NumParsers=0,nginx.Options.ConfigFile=,nginx.Options.LogFormatName=,nginx.Options.TimeFieldName=,nginx.Options.TimeFieldFormat=,nginx.Options.NumParsers=0,postgresql.Options.LogLinePrefix=,regex.Options.LineRegex=[],regex.Options.TimeFieldName=,regex.Options.TimeFieldFormat=,regex.Options.NumParsers=0,syslog.Options.Mode=rfc3164,syslog.Options.ProcessList=,syslog.Options.NumParsers=0
INFO[0000] Starting honeytail
INFO[0000] Running in backfill mode may result in rate-limited events for this dataset. This is expected behavior.
Be aware that if you are also sending data from other sources to this dataset, this may result in events
being dropped.
DEBU[0000] about to call tail.TailFile                   conf="{[msgs.log] [] 0 {beginning true false  false}}" location="<nil>" statefile=/tmp/msgs.leash.state tailConf="{<nil> false true false false <nil> false 0 0xc0000a7230}"
DEBU[0000] attempting to process line                    line="<0>Dec 16 15:14:12 ftp17 dbus[1856970]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service'"
DEBU[0000] Initializing stats reporting. Will print stats once/60 seconds
DEBU[0000] attempting to process line                    line="<0>Dec 16 15:14:12 ftp17 dbus[1856970]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.login1.service': Unit is masked."
DEBU[0000] lines channel is closed, ending syslog processor
DEBU[0000] event send record received                    body= duration=2.344894ms error="<nil>" retry_send=false status_code=202 timestamp="2024-12-16 15:14:12 +0000 UTC"
DEBU[0000] event send record received                    body= duration=2.344894ms error="<nil>" retry_send=false status_code=202 timestamp="2024-12-16 15:14:12 +0000 UTC"
INFO[0000] Summary of sent events                        avg_duration=2.344894ms count=2 count_per_status="map[202:2]" errors="map[]" fastest=2.344894ms lifetime_count=2 response_bodies="map[:2]" slowest=2.344894ms
INFO[0000] Last parsed event                             event="map[content:[system] Activation via systemd failed for unit 'dbus-org.freedesktop.login1.service': Unit is masked. facility:0 hostname:ftp17 priority:0 process:dbus severity:0 timestamp:2024-12-16 15:14:12 +0000 UTC]" event_timestamp="2024-12-16 15:14:12 +0000 UTC"
INFO[0000] Total number of events sent                   number sent by response status code="map[202:2]" total attempted sends=2
INFO[0000] Honeytail is all done, goodbye!

Versions

• Go: go version go1.22.7 linux/amd64
• Honeytail: 1.10.0

Steps to reproduce

Set the system timezone to "America/Chicago"
Run honeytail with --localtime or --timezone="America/Chicago"

Additional context
Running on Amazon Linux 2, so I had to compile the binary for a compatible glibc.