filter_lua: support enable_flb_null #7906

nokute78 · 2023-09-08T22:49:28Z

#7708 #7799 (comment)
In Lua, nil value is a special value to delete key/value pair from associative array.
https://www.lua.org/pil/2.5.html

you can assign nil to a table field to delete it.

It means the key/value will be removed from record if its value is null.
e.g. If record is {"aaa":null, "bbb":"cccc"} ,"aaa":null will be removed in Lua.

This patch is to support enable_flb_null.
If set, fluent-bit pass a lightuserdata flb_null as a nil value to prevent removing key/value.

Default value is false to keep backward compatibility.

Value	Description	Default
`enable_flb_null`	If enabled, null will be converted to flb_null in Lua. It is useful to prevent removing key/value since nil is a special value to remove key value from map in Lua.	false

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

Backporting

Backport to latest stable release.

Configuration

[INPUT]
    name dummy
    dummy {"null":null}

[FILTER]
    name lua
    match *
    call append_null
    code function append_null(tag, timestamp, record) new_record = record new_record["lua_null"] = flb_null return 1, timestamp, new_record end
    enable_flb_null true

[OUTPUT]
    name stdout
    match *

Debug/Valgrind output

$ valgrind --leak-check=full bin/fluent-bit -c a.conf 
==3175== Memcheck, a memory error detector
==3175== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3175== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==3175== Command: bin/fluent-bit -c a.conf
==3175== 
Fluent Bit v2.1.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/09/09 07:47:20] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3175
[2023/09/09 07:47:21] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:47:21] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:47:21] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:47:21] [ info] [input:dummy:dummy.0] initializing
[2023/09/09 07:47:21] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/09/09 07:47:21] [ info] [flb_lua_enable_flb_null] set flb_null
[2023/09/09 07:47:21] [ info] [output:stdout:stdout.0] worker #0 started
[2023/09/09 07:47:21] [ info] [sp] stream processor started
[0] dummy.0: [[1694213242.034084558, {}], {"null"=>nil, "lua_null"=>nil}]
[0] dummy.0: [[1694213243.033679723, {}], {"null"=>nil, "lua_null"=>nil}]
^C[2023/09/09 07:47:24] [engine] caught signal (SIGINT)
[2023/09/09 07:47:24] [ warn] [engine] service will shutdown in max 5 seconds
[0] dummy.0: [[1694213244.005583047, {}], {"null"=>nil, "lua_null"=>nil}]
[2023/09/09 07:47:24] [ info] [input] pausing dummy.0
[2023/09/09 07:47:25] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/09 07:47:25] [ info] [input] pausing dummy.0
[2023/09/09 07:47:25] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/09/09 07:47:25] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==3175== 
==3175== HEAP SUMMARY:
==3175==     in use at exit: 0 bytes in 0 blocks
==3175==   total heap usage: 1,826 allocs, 1,826 frees, 1,636,127 bytes allocated
==3175== 
==3175== All heap blocks were freed -- no leaks are possible
==3175== 
==3175== For lists of detected and suppressed errors, rerun with: -s
==3175== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

$ valgrind --leak-check=full bin/flb-rt-filter_lua 
==3184== Memcheck, a memory error detector
==3184== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3184== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==3184== Command: bin/flb-rt-filter_lua
==3184== 
Test hello_world...                             [2023/09/09 07:48:23] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3184
[2023/09/09 07:48:24] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:48:24] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:48:24] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:48:24] [ info] [input:dummy:dummy.0] initializing
[2023/09/09 07:48:24] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/09/09 07:48:24] [ info] [output:stdout:stdout.0] worker #0 started
[2023/09/09 07:48:24] [ info] [sp] stream processor started
[2023/09/09 07:48:24] [ warn] [engine] service will shutdown in max 1 seconds
[2023/09/09 07:48:24] [ info] [input] pausing dummy.0
[2023/09/09 07:48:25] [ info] [engine] service has stopped (0 pending tasks)
[2023/09/09 07:48:25] [ info] [input] pausing dummy.0
[2023/09/09 07:48:25] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/09/09 07:48:25] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[ OK ]
Test append_tag...                              [2023/09/09 07:48:25] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3184
[2023/09/09 07:48:25] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:48:25] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:48:25] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:48:25] [ info] [input:lib:lib.0] initializing
[2023/09/09 07:48:25] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/09/09 07:48:25] [ info] [sp] stream processor started
[2023/09/09 07:48:26] [ warn] [engine] service will shutdown in max 1 seconds
[2023/09/09 07:48:27] [ info] [engine] service has stopped (0 pending tasks)
[ OK ]
Test type_int_key...                            [2023/09/09 07:48:27] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3184
[2023/09/09 07:48:27] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:48:27] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:48:27] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:48:27] [ info] [input:lib:lib.0] initializing
[2023/09/09 07:48:27] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/09/09 07:48:27] [ info] [sp] stream processor started
[2023/09/09 07:48:28] [ warn] [engine] service will shutdown in max 1 seconds
[2023/09/09 07:48:29] [ info] [engine] service has stopped (0 pending tasks)
[ OK ]
Test type_int_key_multi...                      [2023/09/09 07:48:29] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3184
[2023/09/09 07:48:29] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:48:29] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:48:29] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:48:29] [ info] [input:lib:lib.0] initializing
[2023/09/09 07:48:29] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/09/09 07:48:29] [ info] [sp] stream processor started
[2023/09/09 07:48:30] [ warn] [engine] service will shutdown in max 1 seconds
[2023/09/09 07:48:31] [ info] [engine] service has stopped (0 pending tasks)
[ OK ]
Test type_array_key...                          [2023/09/09 07:48:31] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3184
[2023/09/09 07:48:31] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:48:31] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:48:31] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:48:31] [ info] [input:lib:lib.0] initializing
[2023/09/09 07:48:31] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/09/09 07:48:31] [ info] [sp] stream processor started
[2023/09/09 07:48:32] [ warn] [engine] service will shutdown in max 1 seconds
[2023/09/09 07:48:33] [ info] [engine] service has stopped (0 pending tasks)
[ OK ]
Test array_contains_null...                     [2023/09/09 07:48:33] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3184
[2023/09/09 07:48:33] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:48:33] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:48:33] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:48:33] [ info] [input:lib:lib.0] initializing
[2023/09/09 07:48:33] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/09/09 07:48:33] [ info] [sp] stream processor started
[2023/09/09 07:48:34] [ warn] [engine] service will shutdown in max 1 seconds
[2023/09/09 07:48:35] [ info] [engine] service has stopped (0 pending tasks)
[ OK ]
Test drop_all_records...                        [2023/09/09 07:48:35] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3184
[2023/09/09 07:48:35] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:48:35] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:48:35] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:48:35] [ info] [input:lib:lib.0] initializing
[2023/09/09 07:48:35] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/09/09 07:48:35] [ info] [sp] stream processor started
[2023/09/09 07:48:36] [ warn] [engine] service will shutdown in max 1 seconds
[2023/09/09 07:48:37] [ info] [engine] service has stopped (0 pending tasks)
[ OK ]
Test enable_flb_null...                         [2023/09/09 07:48:37] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3184
[2023/09/09 07:48:37] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:48:37] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:48:37] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:48:37] [ info] [input:lib:lib.0] initializing
[2023/09/09 07:48:37] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/09/09 07:48:37] [ info] [flb_lua_enable_flb_null] set flb_null
[2023/09/09 07:48:37] [ info] [sp] stream processor started
[2023/09/09 07:48:38] [ warn] [engine] service will shutdown in max 1 seconds
[2023/09/09 07:48:39] [ info] [engine] service has stopped (0 pending tasks)
[ OK ]
Test split_record...                            [2023/09/09 07:48:39] [ info] [fluent bit] version=2.1.9, commit=f7731f3f81, pid=3184
[2023/09/09 07:48:39] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/09/09 07:48:39] [ info] [cmetrics] version=0.6.3
[2023/09/09 07:48:39] [ info] [ctraces ] version=0.3.1
[2023/09/09 07:48:39] [ info] [input:lib:lib.0] initializing
[2023/09/09 07:48:39] [ info] [input:lib:lib.0] storage_strategy='memory' (memory only)
[2023/09/09 07:48:39] [ info] [sp] stream processor started
[2023/09/09 07:48:41] [ warn] [timeout] elapsed_time: 2006
[2023/09/09 07:48:41] [ warn] [engine] service will shutdown in max 1 seconds
[2023/09/09 07:48:42] [ info] [engine] service has stopped (0 pending tasks)
[ OK ]
SUCCESS: All unit tests have passed.
==3184== 
==3184== HEAP SUMMARY:
==3184==     in use at exit: 0 bytes in 0 blocks
==3184==   total heap usage: 14,138 allocs, 14,138 frees, 6,744,973 bytes allocated
==3184== 
==3184== All heap blocks were freed -- no leaks are possible
==3184== 
==3184== For lists of detected and suppressed errors, rerun with: -s
==3184== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Signed-off-by: Takahiro Yamashita <[email protected]>

nokute78 · 2023-09-08T22:49:48Z

Cc @tarruda

tarruda

LGTM

Implementation simpler than I thought, thanks for doing this @nokute78

nokute78 · 2023-09-24T01:19:13Z

Doc PR is fluent/fluent-bit-docs#1205 .

nokute78 added 3 commits September 9, 2023 07:39

lua: push flb_null as a nil value

758da4d

Signed-off-by: Takahiro Yamashita <[email protected]>

tests: runtime: lua: test for enable_flb_null

b5cb732

Signed-off-by: Takahiro Yamashita <[email protected]>

filter_lua: support enable_flb_null

421ca08

Signed-off-by: Takahiro Yamashita <[email protected]>

nokute78 requested review from edsiper, leonardo-albertovich, fujimotos and koleini as code owners September 8, 2023 22:49

github-actions bot added the docs-required label Sep 8, 2023

nokute78 mentioned this pull request Sep 8, 2023

filter_lua: support nil_str to replace null value by nil_str string #7799

Closed

4 tasks

nokute78 temporarily deployed to pr September 8, 2023 22:50 — with GitHub Actions Inactive

nokute78 temporarily deployed to pr September 8, 2023 23:21 — with GitHub Actions Inactive

tarruda approved these changes Sep 10, 2023

View reviewed changes

edsiper merged commit e139f6c into fluent:master Sep 23, 2023
42 checks passed

nokute78 deleted the lua_null_lightuserdata branch September 24, 2023 00:51

nokute78 mentioned this pull request Sep 24, 2023

pipeline: filters: lua: add enable_flb_null fluent/fluent-bit-docs#1205

Merged

Porkepix mentioned this pull request Sep 28, 2023

fluent-bit 2.1.10 Homebrew/homebrew-core#146545

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filter_lua: support enable_flb_null #7906

filter_lua: support enable_flb_null #7906

nokute78 commented Sep 8, 2023

nokute78 commented Sep 8, 2023

tarruda left a comment

nokute78 commented Sep 24, 2023

filter_lua: support enable_flb_null #7906

filter_lua: support enable_flb_null #7906

Conversation

nokute78 commented Sep 8, 2023

Configuration

Debug/Valgrind output

nokute78 commented Sep 8, 2023

tarruda left a comment

Choose a reason for hiding this comment

nokute78 commented Sep 24, 2023