Log shipping to Loki broke on upgrading fluent-bit v2.1.10 --> v2.2.0

[theogaraj]

Bug Report

Describe the bug
After recently (Dec 2, 2023) upgrading from fluent-bit v2.1.10 to v2.2.0, logs are no longer being received by Loki (v2.8.2).

To Reproduce

• Install Loki v2.8.2 on an AWS EC2 Linux instance.
• Install fluent-bit v2.1.10 on a different AWS EC2 Linux instance.
• Configure fluent-bit with INPUT as tail of a log file, and OUTPUT as Loki instance

{"log":"YOUR LOG MESSAGE HERE","stream":"stdout","time":"2018-06-11T14:37:30.681701731Z"}

• Steps to reproduce the problem:
  • upgrade fluent-bit version to v2.2.0 (currently latest version)

Expected behavior
Logs should continue to be shipped to Loki

Your Environment

• Version used:

[ec2-user@ip-172-31-42-216 bin]$ /opt/fluent-bit/bin/fluent-bit --version
Fluent Bit v2.1.10

loki, version 2.8.2 (branch: HEAD, revision: 9f809eda7)
  build user:       root@b7e9ca0bf6e0
  build date:       2023-05-03T11:13:57Z
  go version:       go1.20.4
  platform:         linux/arm64

• Configuration:

[SERVICE]
    # Flush
    # =====
    # set an interval of seconds before to flush records to a destination
    flush        1

    # Daemon
    # ======
    # instruct Fluent Bit to run in foreground or background mode.
    daemon       Off

    # Log_Level
    # =========
    # Set the verbosity level of the service, values can be:
    #
    # - error
    # - warning
    # - info
    # - debug
    # - trace
    #
    # by default 'info' is set, that means it includes 'error' and 'warning'.
    log_level    info

    # Parsers File
    # ============
    # specify an optional 'Parsers' configuration file
    parsers_file parsers.conf

    # Plugins File
    # ============
    # specify an optional 'Plugins' configuration file to load external plugins.
    plugins_file plugins.conf

    # HTTP Server
    # ===========
    # Enable/Disable the built-in HTTP Server for metrics
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020

    # Storage
    # =======
    # Fluent Bit can use memory and filesystem buffering based mechanisms
    #
    # - https://docs.fluentbit.io/manual/administration/buffering-and-storage
    #
    # storage metrics
    # ---------------
    # publish storage pipeline metrics in '/api/v1/storage'. The metrics are
    # exported only if the 'http_server' option is enabled.
    #
    storage.metrics on

    # storage.path
    # ------------
    # absolute file system path to store filesystem data buffers (chunks).
    #
    # storage.path /tmp/storage

    # storage.sync
    # ------------
    # configure the synchronization mode used to store the data into the
    # filesystem. It can take the values normal or full.
    #
    # storage.sync normal

    # storage.checksum
    # ----------------
    # enable the data integrity check when writing and reading data from the
    # filesystem. The storage layer uses the CRC32 algorithm.
    #
    # storage.checksum off

    # storage.backlog.mem_limit
    # -------------------------
    # if storage.path is set, Fluent Bit will look for data chunks that were
    # not delivered and are still in the storage layer, these are called
    # backlog data. This option configure a hint of maximum value of memory
    # to use when processing these records.
    #
    # storage.backlog.mem_limit 5M

[INPUT]
    name    tail
    path    /home/ec2-user/worker*/logs/all_logs.txt
    parser  ins_log_parser


[OUTPUT]
    name    loki
    host    ip-xxx-xxx-xxx-xxx.us-east-2.compute.internal
    port    3100
    labels  app=mylabel
    match   *

• Server type and version: AWS EC2 t4g.micro instances
• Operating System and version: Amazon Linux 2
• Filters and plugins:

Additional context
I do see the logs in my terminal if I change my output configuration from loki to stdout, so tailing my log files does not seem to be the problem. I don't know enough to debug deeper than this.

After downgrading back to v2.1.10 on one of my instances, everything works as expected. I'm not sure what was changed between the two versions (maybe just some changes to required configuration?) that resulted in things breaking.

Happy to provide additional info that might be helpful.

[patrick-stephens]

I think this is a duplicate, I'm sure there's a few others so let me see if I can find them.

#8184 and the fix should be in next release I think from #8223

[theogaraj]

Excellent, thanks for the quick fix!

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[theogaraj]

Closing, this is no longer an issue with the latest version.