in_snmp: an input plugin for collect metrics by SNMP request #7671

k402xxxcenxxx · 2023-07-09T15:42:39Z

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change

[INPUT]
    Name snmp
    Tag snmp.test
    Target_Host 172.17.0.3
    Community public
    Oid_Type get
    Oid 1.3.6.1.2.1.1.3.0
    Timeout 1
    Retries 20
[OUTPUT]
    Name stdout
    Match *

Debug log output from testing the change

$ valgrind --leak-check=full --show-leak-kinds=all ./bin/fluent-bit -c in_snmp.conf  -v
==26== Memcheck, a memory error detector
==26== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==26== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==26== Command: ./bin/fluent-bit -c in_snmp.conf -v
==26== 
Fluent Bit v2.1.7
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/11 06:00:49] [ info] Configuration:
[2023/07/11 06:00:49] [ info]  flush time     | 1.000000 seconds
[2023/07/11 06:00:49] [ info]  grace          | 5 seconds
[2023/07/11 06:00:49] [ info]  daemon         | 0
[2023/07/11 06:00:49] [ info] ___________
[2023/07/11 06:00:49] [ info]  inputs:
[2023/07/11 06:00:49] [ info]      snmp
[2023/07/11 06:00:49] [ info] ___________
[2023/07/11 06:00:49] [ info]  filters:
[2023/07/11 06:00:49] [ info] ___________
[2023/07/11 06:00:49] [ info]  outputs:
[2023/07/11 06:00:49] [ info]      stdout.0
[2023/07/11 06:00:49] [ info] ___________
[2023/07/11 06:00:49] [ info]  collectors:
[2023/07/11 06:00:49] [ info] [fluent bit] version=2.1.7, commit=7182681f78, pid=26
[2023/07/11 06:00:49] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/07/11 06:00:49] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/11 06:00:49] [ info] [cmetrics] version=0.6.3
[2023/07/11 06:00:49] [ info] [ctraces ] version=0.3.1
[2023/07/11 06:00:49] [ info] [input:snmp:snmp.0] initializing
[2023/07/11 06:00:49] [ info] [input:snmp:snmp.0] storage_strategy='memory' (memory only)
[2023/07/11 06:00:49] [debug] [snmp:snmp.0] created event channels: read=21 write=22
[2023/07/11 06:00:49] [debug] [stdout:stdout.0] created event channels: read=23 write=24
[2023/07/11 06:00:49] [ info] [sp] stream processor started
[2023/07/11 06:00:49] [ info] [output:stdout:stdout.0] worker #0 started
[2023/07/11 06:00:50] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:51] [debug] [task] created task=0x54cc990 id=0 OK
[2023/07/11 06:00:51] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] snmp.test: [[1689055250.198336300, {}], {"iso.3.6.1.2.1.1.3.0"=>"57"}]
[2023/07/11 06:00:51] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:51] [debug] [out flush] cb_destroy coro_id=0
[2023/07/11 06:00:51] [debug] [task] destroy task=0x54cc990 (task_id=0)
[2023/07/11 06:00:52] [debug] [task] created task=0x5532f10 id=0 OK
[2023/07/11 06:00:52] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] snmp.test: [[1689055251.182227200, {}], {"iso.3.6.1.2.1.1.3.0"=>"58"}]
[2023/07/11 06:00:52] [debug] [out flush] cb_destroy coro_id=1
[2023/07/11 06:00:52] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:52] [debug] [task] destroy task=0x5532f10 (task_id=0)
[2023/07/11 06:00:53] [debug] [task] created task=0x5598fd0 id=0 OK
[2023/07/11 06:00:53] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] snmp.test: [[1689055252.137364100, {}], {"iso.3.6.1.2.1.1.3.0"=>"59"}]
[2023/07/11 06:00:53] [debug] [out flush] cb_destroy coro_id=2
[2023/07/11 06:00:53] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:53] [debug] [task] destroy task=0x5598fd0 (task_id=0)
[2023/07/11 06:00:54] [debug] [task] created task=0x56010d0 id=0 OK
[0] snmp.test: [[1689055253.136002200, {}], {"iso.3.6.1.2.1.1.3.0"=>"60"}]
[2023/07/11 06:00:54] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/07/11 06:00:54] [debug] [out flush] cb_destroy coro_id=3
[2023/07/11 06:00:54] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:54] [debug] [task] destroy task=0x56010d0 (task_id=0)
^C[2023/07/11 06:00:54] [engine] caught signal (SIGINT)
[2023/07/11 06:00:54] [debug] [task] created task=0x56671e0 id=0 OK
[2023/07/11 06:00:54] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] snmp.test: [[1689055254.149898200, {}], {"iso.3.6.1.2.1.1.3.0"=>"61"}]
[2023/07/11 06:00:54] [ warn] [engine] service will shutdown in max 5 seconds
[2023/07/11 06:00:54] [ info] [input] pausing snmp.0
[2023/07/11 06:00:54] [debug] [out flush] cb_destroy coro_id=4
[2023/07/11 06:00:54] [debug] [task] destroy task=0x56671e0 (task_id=0)
[2023/07/11 06:00:55] [ info] [engine] service has stopped (0 pending tasks)
read_config_store open failure on /var/lib/snmp/snmp.conf
[2023/07/11 06:00:55] [ info] [input] pausing snmp.0
read_config_store open failure on /var/lib/snmp/snmp.conf
read_config_store open failure on /var/lib/snmp/snmp.conf
[2023/07/11 06:00:55] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/11 06:00:55] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==26== 
==26== HEAP SUMMARY:
==26==     in use at exit: 4,800 bytes in 1 blocks
==26==   total heap usage: 3,723 allocs, 3,722 frees, 2,543,883 bytes allocated
==26== 
==26== 4,800 bytes in 1 blocks are still reachable in loss record 1 of 1
==26==    at 0x483AB65: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==26==    by 0x4AE51F7: netsnmp_init_mib_internals (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x4AE9C8B: ??? (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x4AFB5F4: init_snmp (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x42BBD6: in_snmp_init (in_snmp.c:372)
==26==    by 0x1FC78D: flb_input_instance_init (flb_input.c:1189)
==26==    by 0x1FC925: flb_input_init_all (flb_input.c:1248)
==26==    by 0x23D4B0: flb_engine_start (flb_engine.c:730)
==26==    by 0x1DED52: flb_lib_worker (flb_lib.c:638)
==26==    by 0x4855EA6: start_thread (pthread_create.c:477)
==26==    by 0x5069A2E: clone (clone.S:95)
==26== 
==26== LEAK SUMMARY:
==26==    definitely lost: 0 bytes in 0 blocks
==26==    indirectly lost: 0 bytes in 0 blocks
==26==      possibly lost: 0 bytes in 0 blocks
==26==    still reachable: 4,800 bytes in 1 blocks
==26==         suppressed: 0 bytes in 0 blocks
==26== 
==26== For lists of detected and suppressed errors, rerun with: -s
==26== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0

Attached Valgrind output that shows no leaks or memory corruption was found

==26== 
==26== HEAP SUMMARY:
==26==     in use at exit: 4,800 bytes in 1 blocks
==26==   total heap usage: 3,723 allocs, 3,722 frees, 2,543,883 bytes allocated
==26== 
==26== 4,800 bytes in 1 blocks are still reachable in loss record 1 of 1
==26==    at 0x483AB65: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==26==    by 0x4AE51F7: netsnmp_init_mib_internals (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x4AE9C8B: ??? (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x4AFB5F4: init_snmp (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x42BBD6: in_snmp_init (in_snmp.c:372)
==26==    by 0x1FC78D: flb_input_instance_init (flb_input.c:1189)
==26==    by 0x1FC925: flb_input_init_all (flb_input.c:1248)
==26==    by 0x23D4B0: flb_engine_start (flb_engine.c:730)
==26==    by 0x1DED52: flb_lib_worker (flb_lib.c:638)
==26==    by 0x4855EA6: start_thread (pthread_create.c:477)
==26==    by 0x5069A2E: clone (clone.S:95)
==26== 
==26== LEAK SUMMARY:
==26==    definitely lost: 0 bytes in 0 blocks
==26==    indirectly lost: 0 bytes in 0 blocks
==26==      possibly lost: 0 bytes in 0 blocks
==26==    still reachable: 4,800 bytes in 1 blocks
==26==         suppressed: 0 bytes in 0 blocks
==26== 
==26== For lists of detected and suppressed errors, rerun with: -s
==26== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0

There is one leak and can be fixed after this patch is release: net-snmp/net-snmp@4bd0d9a
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

fluent/fluent-bit-docs#1162
Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

k402xxxcenxxx · 2023-07-15T07:23:03Z

I have found a leak in the code, and it can be resolved by applying this patch: net-snmp/net-snmp@4bd0d9a.

However, I am unsure about the process of incorporating this fix into the fluent-bit project. Any suggestions or guidance on this matter would be greatly appreciated!

patrick-stephens

I've triggered a full build of all targets to confirm it builds for them, we probably need to modify dependencies to build.

Can you confirm it works for non-Linux targets as well? It may build but that doesn't mean it works...

patrick-stephens · 2023-07-17T09:10:35Z

CMakeLists.txt

@@ -191,6 +192,7 @@ option(FLB_IN_HEAD                     "Enable Head input plugin"
 option(FLB_IN_PROC                     "Enable Process input plugin"                  Yes)
 option(FLB_IN_SYSTEMD                  "Enable Systemd input plugin"                  Yes)
 option(FLB_IN_DUMMY                    "Enable Dummy input plugin"                    Yes)
+option(FLB_IN_SNMP                     "Enable SNMP input plugin"                     Yes)


To confirm - does this work on Windows and macOS as well? It will need disabling if not in their specific override cmake/ files.

https://github.com/fluent/fluent-bit/blob/master/cmake/macros.cmake

https://github.com/fluent/fluent-bit/blob/master/cmake/windows-setup.cmake

patrick-stephens · 2023-07-17T09:11:07Z

dockerfiles/Dockerfile

@@ -53,6 +53,7 @@ RUN echo "deb http://deb.debian.org/debian bullseye-backports main" >> /etc/apt/
    flex \
    bison \
    libyaml-dev \
+    libsnmp-dev \


We presumably will also need this in the various package builds under packaging/distros too.

k402xxxcenxxx · 2023-07-18T04:23:43Z

@patrick-stephens thanks, I will try to build and run on Windows device

k402xxxcenxxx · 2023-08-18T06:02:39Z

I still fail to build it on Windows, so I choose to disable it in https://github.com/fluent/fluent-bit/blob/master/cmake/windows-setup.cmake.
And I'm trying to build to various distribution

github-actions · 2023-12-11T01:51:17Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

davidvasandani · 2024-11-19T19:14:12Z

@k402xxxcenxxx What's needed to finish this PR? Do you need help testing on a Windows device?

k402xxxcenxxx · 2024-12-11T06:34:05Z

@k402xxxcenxxx What's needed to finish this PR? Do you need help testing on a Windows device?

Hi @davidvasandani ,

Thanks for asking. I've been too busy lately to maintain this PR and I'm also considering not supporting this on Windows.
I'll try to handle on this PR as soon as possible.

Thanks again for your kind reminding.

Add: ChatGPT in_snmp Add: snmp get with fix request Add: snmp with log recorder Add: print value by sprint_realloc_by_type Fix: refine code. Add: walk get values TMP Signed-off-by: k402xxxcenxxx <[email protected]>

Add: unit test for snmp_walk Signed-off-by: k402xxxcenxxx <[email protected]>

Signed-off-by: k402xxxcenxxx <[email protected]>

github-actions bot added the docs-required label Jul 9, 2023

k402xxxcenxxx changed the title ~~Feature/in snmp~~ in_snmp: an input plugin for collect metrics by SNMP request Jul 15, 2023

k402xxxcenxxx force-pushed the feature/in_snmp branch from 869e75c to eb87261 Compare July 15, 2023 06:09

k402xxxcenxxx mentioned this pull request Jul 15, 2023

in_snmp: Add documentation for input SNMP plugin fluent/fluent-bit-docs#1162

Open

k402xxxcenxxx marked this pull request as ready for review July 15, 2023 07:17

k402xxxcenxxx requested review from niedbalski, patrick-stephens, fujimotos, edsiper, leonardo-albertovich and koleini as code owners July 15, 2023 07:17

patrick-stephens added the ok-package-test Run PR packaging tests label Jul 17, 2023

patrick-stephens reviewed Jul 17, 2023

View reviewed changes

k402xxxcenxxx had a problem deploying to pr July 17, 2023 09:15 — with GitHub Actions Failure

k402xxxcenxxx force-pushed the feature/in_snmp branch from eb87261 to 02ae8c0 Compare July 21, 2023 08:53

k402xxxcenxxx had a problem deploying to pr July 24, 2023 09:35 — with GitHub Actions Failure

k402xxxcenxxx force-pushed the feature/in_snmp branch from 02ae8c0 to 63597e4 Compare August 9, 2023 03:43

k402xxxcenxxx requested a review from celalettin1286 as a code owner August 9, 2023 03:43

k402xxxcenxxx force-pushed the feature/in_snmp branch 2 times, most recently from 265dab6 to 1c78626 Compare August 18, 2023 06:00

k402xxxcenxxx temporarily deployed to pr August 21, 2023 10:22 — with GitHub Actions Inactive

k402xxxcenxxx temporarily deployed to pr August 21, 2023 10:53 — with GitHub Actions Inactive

github-actions bot added the Stale label Dec 11, 2023

github-actions bot removed the Stale label Aug 16, 2024

k402xxxcenxxx added 6 commits December 15, 2024 02:36

snmp: borrow dummy plugin as a start point

cf8b9ff

Add: ChatGPT in_snmp Add: snmp get with fix request Add: snmp with log recorder Add: print value by sprint_realloc_by_type Fix: refine code. Add: walk get values TMP Signed-off-by: k402xxxcenxxx <[email protected]>

Add: unit test

014716a

Add: unit test for snmp_walk Signed-off-by: k402xxxcenxxx <[email protected]>

Fix: memory leak due to openssl lib used by net-snmp

ae52940

Signed-off-by: k402xxxcenxxx <[email protected]>

Add: timeout and retris setting for net snmp lib

5602f0d

Signed-off-by: k402xxxcenxxx <[email protected]>

Add: snmp in Dockerfile

33f3555

Signed-off-by: k402xxxcenxxx <[email protected]>

Fix: not support Windows. add libsnmp in various Dockerfile

9a4ed5a

Signed-off-by: k402xxxcenxxx <[email protected]>

k402xxxcenxxx force-pushed the feature/in_snmp branch from 1c78626 to 9a4ed5a Compare December 15, 2024 02:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

in_snmp: an input plugin for collect metrics by SNMP request #7671

in_snmp: an input plugin for collect metrics by SNMP request #7671

k402xxxcenxxx commented Jul 9, 2023 •

edited

Loading

k402xxxcenxxx commented Jul 15, 2023 •

edited

Loading

patrick-stephens left a comment

patrick-stephens Jul 17, 2023

patrick-stephens Jul 17, 2023

k402xxxcenxxx commented Jul 18, 2023

k402xxxcenxxx commented Aug 18, 2023

github-actions bot commented Dec 11, 2023

davidvasandani commented Nov 19, 2024

k402xxxcenxxx commented Dec 11, 2024

in_snmp: an input plugin for collect metrics by SNMP request #7671

Are you sure you want to change the base?

in_snmp: an input plugin for collect metrics by SNMP request #7671

Conversation

k402xxxcenxxx commented Jul 9, 2023 • edited Loading

k402xxxcenxxx commented Jul 15, 2023 • edited Loading

patrick-stephens left a comment

Choose a reason for hiding this comment

patrick-stephens Jul 17, 2023

Choose a reason for hiding this comment

patrick-stephens Jul 17, 2023

Choose a reason for hiding this comment

k402xxxcenxxx commented Jul 18, 2023

k402xxxcenxxx commented Aug 18, 2023

github-actions bot commented Dec 11, 2023

davidvasandani commented Nov 19, 2024

k402xxxcenxxx commented Dec 11, 2024

k402xxxcenxxx commented Jul 9, 2023 •

edited

Loading

k402xxxcenxxx commented Jul 15, 2023 •

edited

Loading