Make hot reloadable

[cosmo0920]

This PR needs to work with fluent/fluent-bit#7997.

• Bind pre_run callbacks to ensure only once execution at plugin lifecycles.
• Bind pause/resume callbacks to synchronize ordinary lifecycles of input plugins.
• Add handling for Fluent Bit V2 formatted events
• pre_exit callbacks are not binded for now. However, the PR plugin_proxy: core: expose more callbacks for golang plugins fluent/fluent-bit#7997 provides such callbacks.

[niedbalski]

@nicolasparada can you take a look at this one please?

[pwhelan]

IWFM! I'll do another test once #52 lands as well.

[cosmo0920]

IWFM! I'll do another test once #52 lands as well.

I've been suffering for fatal more stack g0 error. Did you use Golang 1.20 or 1.21?
I got the error with Golang 1.12.1...

[cosmo0920]

I confirmed that we needn't use guard for absent channels with #52 mechanism. No need to change the API usages on derived plugins.

[pwhelan]

IWFM! I'll do another test once #52 lands as well.

I've been suffering for fatal more stack g0 error. Did you use Golang 1.20 or 1.21? I got the error with Golang 1.12.1...

I used go 1.20 per your instruction. From the materials you sent me it's clear 1.21 will not work.

[cosmo0920]

Ok, I got it

[niedbalski]

LGTM

[cosmo0920]

@pwhelan Any thoughts?

[pwhelan]

IWFM!

I tested it with the http_loader plugin and master fluent-bit and it seems to at least work as well as before.

./fluent-bit-master -c crash-local.conf 
Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/10/20 12:30:39] [ info] [fluent bit] version=2.2.0, commit=bf17059bf6, pid=74476
[2023/10/20 12:30:39] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/10/20 12:30:39] [ info] [cmetrics] version=0.6.4
[2023/10/20 12:30:39] [ info] [ctraces ] version=0.3.1
[2023/10/20 12:30:39] [ info] [input:http_loader:http_loader.0] initializing
[2023/10/20 12:30:39] [ info] [input:http_loader:http_loader.0] storage_strategy='memory' (memory only)
[2023/10/20 12:30:39] [debug] Using storage key "http_loader_v2_678d5cf4"
[2023/10/20 12:30:39] [debug] Data loaded from local disk storage with key "http_loader_v2_678d5cf4"
[2023/10/20 12:30:39] [debug] Successfully json unmarshalled loaded data https://ggy66547.live.dynatrace.com/api/v2/auditlogs?nextPageKey=vu8y3hPZ3q0AAAAAAloaRWk33_UAAAGKlw7-6AAAAYqYWJXoAAAACgBCZXZlbnRUeXBlKExPR0lOLExPR09VVCxDUkVBVEUsREVMRVRFLEdFVCxQT1NULEdFTkVSQUwsUkVBRCxVUERBVEUpAb7vMt4T2d6t
[2023/10/20 12:30:39] [debug] Failed to json unmarshal stored request body, returning original string ""
[2023/10/20 12:30:39] [debug] Successfully processed loaded data with index index 686
[2023/10/20 12:30:39] [debug] [input:http_loader:http_loader.0] [thread init] initialization OK
[2023/10/20 12:30:39] [ info] [input:http_loader:http_loader.0] thread instance initialized
[2023/10/20 12:30:39] [ info] [output:stdout:stdout.0] worker #0 started
[2023/10/20 12:30:39] [ info] [output:syslog:devo_coud] setup done for localhost:5140 (TLS=on)
[2023/10/20 12:30:39] [ info] [sp] stream processor started
[2023/10/20 12:30:39] [debug] Collect invoked
[2023/10/20 12:30:39] [debug] Sending request: method="GET" url="https://ggy66547.live.dynatrace.com/api/v2/auditlogs?nextPageKey=vu8y3hPZ3q0AAAAAAloaRWk33_8AAAGKlw7-6AAAAYqYWJXoAAAACgBCZXZlbnRUeXBlKExPR0lOLExPR09VVCxDUkVBVEUsREVMRVRFLEdFVCxQT1NULEdFTkVSQUwsUkVBRCxVUERBVEUpAb7vMt4T2d6t" headers=map[Authorization:[Api-Token dt0c01.UGC5MJC2ZXMKX35HL3KRHH4H.VREOSCZVE36BVZ3LKOESI33X44GICB34K43EEJE7E2Y5DQOX4UFOBXAJSVNMIGYI] User-Agent:[Fluent-Bit HTTP Loader Plugin]] body=""
[2023/10/20 12:30:40] [debug] Received response: status_code=401 headers=map[Cache-Control:[no-store, no-cache] Content-Type:[application/json;charset=utf-8] Date:[Fri, 20 Oct 2023 15:30:40 GMT] Pragma:[no-cache] Server:[ruxit server] Server-Timing:[dtRpid;desc="-1092255344", dtSInfo;desc="0"] Set-Cookie:[dtCookie=v_4_srv_12_sn_569FA794E2D7E168621531932430BCF0_perc_100000_ol_0_mul_1_app-3Af6b10dd0df01cfe1_1_rcs-3Acss_0; Path=/; Domain=.dynatrace.com; secure] Strict-Transport-Security:[max-age=31536000;includeSubDomains] Vary:[Accept-Encoding] X-Oneagent-Js-Injection:[true] X-Robots-Tag:[noindex]] body="{\"error\":{\"code\":401,\"message\":\"Token Authentication failed\"}}"
[2023/10/20 12:30:40] [error] Error collecting data: template: pull_interval:1:208: executing "pull_interval" at <$fromtime.Add>: can't evaluate field Add in type string
2023/10/20 12:30:41 http_loader: next URL now: 2023-10-20 15:30:41.597126241 +0000 UTC
2023/10/20 12:30:41 http_loader: next URL totime: 2023-10-20T15:30:41
[2023/10/20 12:30:41] [error] Error collecting data: template: url:1:511: executing "url" at <$fromtime.Format>: can't evaluate field Format in type string
2023/10/20 12:30:42 http_loader: next URL now: 2023-10-20 15:30:42.600503841 +0000 UTC
2023/10/20 12:30:42 http_loader: next URL totime: 2023-10-20T15:30:42
2023/10/20 12:30:42 http_loader: next URL fromtime: 2023-10-20T15:30:41
2023/10/20 12:30:42 http_loader: next URL from: 2023-10-20T15:30:41, to: 2023-10-20T15:30:42
[2023/10/20 12:30:42] [debug] Sending request: method="GET" url="https://ggy66547.live.dynatrace.com/api/v2/auditlogs?filter=eventType%28LOGIN,LOGOUT,CREATE,DELETE,GET,POST,GENERAL,READ,UPDATE%29&from=2023-10-20T15:30:41&to=2023-10-20T15:30:42&pageSize=1&sort=timestamp" headers=map[Authorization:[Api-Token dt0c01.UGC5MJC2ZXMKX35HL3KRHH4H.VREOSCZVE36BVZ3LKOESI33X44GICB34K43EEJE7E2Y5DQOX4UFOBXAJSVNMIGYI] User-Agent:[Fluent-Bit HTTP Loader Plugin]] body=""
[2023/10/20 12:30:42] [debug] Received response: status_code=401 headers=map[Cache-Control:[no-store, no-cache] Content-Type:[application/json;charset=utf-8] Date:[Fri, 20 Oct 2023 15:30:42 GMT] Pragma:[no-cache] Server:[ruxit server] Server-Timing:[dtRpid;desc="1046871211", dtSInfo;desc="0"] Set-Cookie:[dtCookie=v_4_srv_12_sn_94474B2F387D93BA3DD84A63CC24D392_perc_100000_ol_0_mul_1_app-3Af6b10dd0df01cfe1_1_rcs-3Acss_0; Path=/; Domain=.dynatrace.com; secure] Strict-Transport-Security:[max-age=31536000;includeSubDomains] Vary:[Accept-Encoding] X-Oneagent-Js-Injection:[true] X-Robots-Tag:[noindex]] body="{\"error\":{\"code\":401,\"message\":\"Token Authentication failed\"}}"
[2023/10/20 12:30:42] [debug] Initiating store process with key "http_loader_v2_678d5cf4"
[2023/10/20 12:30:42] [debug] Successfully stored data in local disk storage with key "http_loader_v2_678d5cf4"
[2023/10/20 12:30:42] [ warn] Error updating metadata in cloud API with key "http_loader_v2_678d5cf4": cloud api: client not initialized or pipelineID missing
^C[2023/10/20 12:30:43] [engine] caught signal (SIGINT)
[2023/10/20 12:30:43] [ warn] [engine] service will shutdown in max 5 seconds
[2023/10/20 12:30:43] [debug] [input:http_loader:http_loader.0] thread pause instance
2023/10/20 12:30:43 goroutine will be stopping: name="http_loader"
[2023/10/20 12:30:44] [ info] [engine] service has stopped (0 pending tasks)
[2023/10/20 12:30:44] [debug] [input:http_loader:http_loader.0] thread pause instance
[2023/10/20 12:30:44] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/10/20 12:30:44] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/10/20 12:30:44] [debug] [input:http_loader:http_loader.0] thread exit instance

I also seem to be able to successfully reload the instance as well:

[2023/10/20 12:32:37] [engine] caught signal (SIGHUP)
[2023/10/20 12:32:37] [ info] reloading instance pid=74803 tid=0x7ffff6dffac0
[2023/10/20 12:32:37] [ info] [reload] stop everything of the old context
[2023/10/20 12:32:37] [ warn] [engine] service will shutdown when all remaining tasks are flushed
[2023/10/20 12:32:37] [debug] [input:http_loader:http_loader.0] thread pause instance
2023/10/20 12:32:37 goroutine will be stopping: name="http_loader"
[2023/10/20 12:32:38] [ info] [engine] service has stopped (0 pending tasks)
[2023/10/20 12:32:38] [debug] [input:http_loader:http_loader.0] thread pause instance
[2023/10/20 12:32:38] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/10/20 12:32:38] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2023/10/20 12:32:38] [debug] [input:http_loader:http_loader.0] thread exit instance
[2023/10/20 12:32:38] [ info] [reload] start everything
[2023/10/20 12:32:38] [ info] [fluent bit] version=2.2.0, commit=bf17059bf6, pid=74803
[2023/10/20 12:32:38] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/10/20 12:32:38] [ info] [cmetrics] version=0.6.4
[2023/10/20 12:32:38] [ info] [ctraces ] version=0.3.1
[2023/10/20 12:32:38] [ info] [input:http_loader:http_loader.0] initializing
[2023/10/20 12:32:38] [ info] [input:http_loader:http_loader.0] storage_strategy='memory' (memory only)
[2023/10/20 12:32:38] [debug] Using storage key "http_loader_v2_678d5cf4"
[2023/10/20 12:32:38] [debug] Data loaded from local disk storage with key "http_loader_v2_678d5cf4"

Big stamp of approval. This was with go 1.21 BTW.