Use system time offset for local datetimes

[stefk]

The Time_Offset parameter allows to add a fixed UTC offset to local datetimes. However, that offset is subject to changes in countries that observe a daylight saving time, so hardcoding it in a config file may produce incorrect results at some point. This seems like a common problem but I don't see a way to address it with the current configuration options of Fluent Bit. Am I missing something? Or would you consider adding a parameter, for example, to retrieve that offset directly from the system instead of using a fixed one?

[edsiper]

@stefk there is one feature in Fluent Bit to use variables in the configuration files. Looking carefully in the source code I see that feature is not exposed for the parsers.conf which could address the main issue stated here: using an environment variable would be enough.

This will be addressed during Fluent Bit v0.14 development cycle.

[stefk]

@edsiper thanks for the feedback! Aren't environment variables resolved at startup time only though? If that's the case, scheduling a re-start at offset changes would still be necessary.

[scuzhanglei]

@edsiper My version is v1.0.1, I test it and found out it still not support variables in parsers.conf, is it? thanks a lot for your works.

[scuzhanglei]

@edsiper plz review this PR

[danischroeter]

we got the same issue here...
we got logs that are written with the local timezone (CEST) but without timezone specification. Depending on Summer/Winter time that would mean Time_Offset of 2 or 1 hour...

Instead of the proposed solution I think it would make more sense to define a new property like:
Time_DefaultZone that let's us specifiy the timezone for local timestamps. This way fluentbit could read the time and convert it from Time_DefaultZone to UTC. This would also solve the daylightsavingtime issues...

Maybe this method could even be used as a quickfix/hack: is there a way to modify the time field of a record (manually add CEST) and later parse the time?

[agileknight]

I found a workaround that uses the configured timezone, e.g. "Europe/Berlin" at the time of parsing the log entry.

• Use a lua filter to parse the timestamp
• Use a linux distribution that fully supports timezone information (ulibc does not work)
• Configure the timezone to use to parse at the distro level

Example code (tested with version 1.3.6):

parse_ts.lua (placed next to fluent-bit.conf)

function cb_parse_ts(tag, timestamp, record)
  time=record["time"]
   if (time == nil) then
    return -1, 0, 0 -- drop entry instead of throwing error
  end
  pattern="(%d+)-(%d+)-(%d+) (%d+):(%d+):(%d+),(%d+)" -- example: 2020-01-14 17:23:58,634
  year,month,day,hour,minute,second,millisecond=time:match(pattern)
  new_timestamp = os.time({day=day,month=month,year=year,hour=hour,min=minute,sec=second}) + millisecond/1000
  return 1, new_timestamp, record
end

filter in fluent-bit.conf

[FILTER]
    Name                lua
    Match               *
    script              parse_ts.lua
    call                cb_parse_ts

Dockerfile

FROM debian:stretch

ENV FLUENTBIT_VERSION 1.3.6
ENV DEBIAN_FRONTEND noninteractive

RUN  apt-get update && apt-get install -y \
    wget \
    gnupg2 \
    dirmngr \
    apt-transport-https \
    ca-certificates \
  && rm -rf /var/lib/apt/lists/*

RUN wget -qO - https://packages.fluentbit.io/fluentbit.key | apt-key add -
RUN echo "deb https://packages.fluentbit.io/debian/stretch stretch main" >> /etc/apt/sources.list
RUN  apt-get update && apt-get install -y \
    td-agent-bit=$FLUENTBIT_VERSION \
  && rm -rf /var/lib/apt/lists/*

RUN mkdir -p /fluent-bit/bin && \
  ln -s /opt/td-agent-bit/bin/td-agent-bit /fluent-bit/bin/fluent-bit

USER 1000
CMD ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.conf"]

Run container with env set:

TZ=Europe/Berlin

Also requires mounted files:

• /fluent-bit/etc/fluent-bit.conf
• /fluent-bit/etc/parse_ts.lua

[jtackaberry]

I was also bit by this, so +1 for a native solution.

[shqear93]

@edsiper are you going to support a field like Time_DefaultZone ?

[lacksfish]

Is fluentbit gonna address this issue? Ignoring the timezone when parsing to UTC timestamp is a pretty major bug

Time_Key dateTime
Time_Format %Y-%m-%dT%H:%M:%S,%L%z

Is getting parsed without problems, but the timezone is simply dropped and all our timestamps are 2 hours in the future.

We can not use the Time_Offset key as we have summer/winter-time so time offset fluctuates twice a year between 1 and 2 hours.

Troubling issue......

[LionelCons]

+1

[sp0f]

+1

[tvm1]

+1

[petrkr]

Got now same issue with datetime and timezone. I think it's major to support timezone on software which is used to work with date and times in log files.
There is no solution to have all servers/instances in UTC due other applications.

[tvm1]

It indeed is a major issue.

However the solution kindly provided by @agileknight works. Just adding the LUA script and filter for all records fixed the issue for me. All records arrive to output destination in UTC regardless of the device's current timezone and that's what you want.

[petrkr]

LUA is not solution at all.. We trying get rid of all extra packages on systems (ruby, python2) and keep it minimal, we do not want install LUA just because one script. Also it would lead to much worse performace, since every log line need to go through some another preprocessor.

So it is not solution, but only workaround

[tvm1]

Yes, it is a workaround that solves the problem, because there's no other sane way to handle this currently.

You don't need to install LUA, you just have to compile fluent-bit with LUA support, that isn't going to do much to your system (chances are that it might be enabled already if you use some binary distro).

The overhead of using this solution is in fact negligible. I have weak devices like RPI B's emitting a lot of records without any visible load. So that's not a problem.

[fabricev]

Hello there, got hit by this as well, and the workaround provided by @agileknight didn't seem to match exactly my case. I'm using fluentbit 1.7.1, and local logs are using syslog format (without a TZ printed), my logs are in Europe/Zurich timezone and when pushed to ES they are incorrectly indexed in the future. That's how i ended here, understanding that fluentbit couldn't do a local timezone to UTC before pushing to ES.

I simplified the lua script to adjust the timestamp I get from the syslog parser.

function local_timestamp_to_UTC(tag, timestamp, record)
  local d1 = os.date("*t", 0)
  local d2 = os.date("!*t", 0)
  local zone_diff = os.difftime(os.time(d2), os.time(d1))
  new_timestamp = timestamp + zone_diff
  return 1, new_timestamp, record
end

This is the filter

[FILTER]
    Name         lua
    Match        *
    script       /fluent-bit/etc/adjust_ts.lua
    call         local_timestamp_to_UTC

I used the standard fluent/fluent-bit image with env TZ=Europe/Zurich and it did the expected workaround.

[michaltrmac]

A small modification of the @fabricev solution for correct daylight saving time processing

function cb_fix_timestamp(tag, timestamp, record)
    local utcdate   = os.date("!*t", ts)
    local localdate = os.date("*t", ts)
    localdate.isdst = false -- this is the trick
    utc_time_diff = os.difftime(os.time(localdate), os.time(utcdate))
    return 1, timestamp - utc_time_diff, record
end

[fabricev]

@michaltrmac : You're right, we've just verified that this week, your modification is required for DST !

[LionelCons]

FWIW, the given Lua solution works most of the time but not always. At each DST change, there is a one hour window during which the given offset is wrong.

In a CET/CEST environment, try for instance timestamp=1616895030 (30 minutes before 2021 Spring DST change) or timestamp=1635643830 (90 minutes before 2021 Fall DST change).

[blueice123]

I want to change the time sent from fluent bit to ES to local timezone.
I want log transport with minimal overhead, and LUA is not the solution.

What should I do?

[blueice123]

It would be nice to be able to change the timezone of the fluent bit container.

[braydonk]

@edsiper @leonardo-albertovich is there any chance #7994 could be looked at? I have given it a preliminary review.
This has been a long standing problem, and the PR seems like a good solution. If that isn't a desirable solution, how can we go about getting this resolved?

[braydonk]

I have created a PR that will resolve this with a new configuration field on the parser, Time_System_Timezone. The solution does account for daylight savings time.

[AdaptiveStep]

Anyone got any more ideas? I'm struggling with this.

[braydonk]

@edsiper @leonardo-albertovich Checking again if #8164 could get a review.

[braydonk]

The Time_System_Timezone feature is merged and will be in the next Fluent Bit release. The docs will be updated, but TL;DR the Time_System_Timezone config field on [PARSER] will have that parser switch to an alternative time API (mktime) that detects the TZ environment variable on the system and use that if it is not set from the time format. In my tests, it showed resilience to daylight savings time changes.

Glad this issue can finally be closed after all this time!

[Jiale-Fang]

Hi @braydonk, I am using Fluent Bit 3.1.4 and have set Time_System_Timezone to true. My instance is located in the US Eastern timezone, which is currently observing daylight saving time, so the machine time should be 4 hours behind UTC. However, Fluent Bit is incorrectly adding 5 hours to my log timestamps.

[braydonk]

Hi @Jiale-Fang, I don't know the exact mechanics of Get-TimeZone, but judging by that screenshot your timezone on the machine seems to be incorrectly set to standard time. This Fluent Bit feature uses mktime, which respects the `TZ` environment variable. For the feature to work as expected, you should ensure the `TZ` environment variable is set on the machine to the desired time zone.

…

On Fri, Aug 2, 2024, 6:01 PM Jiale Fang ***@***.***> wrote: Hi @braydonk <https://github.com/braydonk>, I am using Fluent Bit 3.1.4 and have set Time_System_Timezone to true. My instance is located in the US Eastern timezone, which is currently observing daylight saving time, so the machine time should be 4 hours behind UTC. However, Fluent Bit is incorrectly adding 5 hours to my log timestamps. image.png (view on web) <https://github.com/user-attachments/assets/60d76369-bbb5-439f-b72f-f2b580cede4b> — Reply to this email directly, view it on GitHub <#593 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWJXJSDY2Q2ZOG4XKNUSHY3ZPP6TFAVCNFSM4FAGKTDKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMRWGYYTSNJQGMYQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Jiale-Fang]

Hi @braydonk ,thank you for your kind response. Now Time Key can use system default time zone. I have another question: I am trying to input logs into OpenSearch using Fluent Bit, but the timezone of the machine running Fluent Bit is set to EDT. For specific reasons, I need the time key in the OpenSearch index to be in UTC. Is there a way to achieve this conversion? (During daylight saving time, the offset from UTC is 4 hours, and during standard time, the offset is 5 hours) I can't find a way to make Time_Offset automatically adjust for daylight saving time.

[edsiper]

thanks everyone, but I think is time to close this issue since the main feature is already available. For any new inquiry please open a new ticket.

[gwharton]

Possible bug in implementation.
#9197