Add bearer token auth for loki #1224

raynay-r · 2024-06-28T21:00:42Z

What this PR does / why we need it:

I was searching for an alternative of the openshift-logging-operator. None of the available operators supported bearer token authentication for the Loki output. Since bearer authentication is very useful together wiith Loki and the openshift oauth proxy I decided to make this contribution.

Which issue(s) this PR fixes:

Fixes #1221 (Not my ticket, by coincidence someone else seems to be missing this feature)

Does this PR introduced a user-facing change?

 * [Fluent-Bit] Additional Parameter "bearerToken" for (Cluster) Output of type Loki
 * [Fluent-Bit] Additional Parameter "uri" for (Cluster) Output of type Loki
 * [Fluentd] Additional Parameter "bearerTokenFile" for ClusterOutput of type Loki

Additional documentation, usage docs, etc.:

- [Usage]: https://docs.fluentbit.io/manual/pipeline/outputs/loki
- [Other doc] https://github.com/fluent/fluent-operator/blob/master/docs/plugins/fluentbit/output/loki.md
- [Usage] https://github.com/fluent/fluent-operator/blob/master/docs/plugins/fluentd/output/loki.md
- [Other doc] https://docs.fluentbit.io/manual/pipeline/outputs/loki

raynay-r · 2024-06-28T21:02:43Z

I had to update the controller-gen version, otherwise the generation step would segfault. This caused a lot of changes. I am not sure how this should be handled.

cw-Guo · 2024-07-01T02:35:58Z

@raynay-r Thanks so much for you first contribution. But upgrade the contrller-gen version might not be a good practice for this MR. Can you please try to fix segfault? Feel free to share the problem you have when generating code here. Let us know the details so that we can help you.

benjaminhuo · 2024-07-01T06:29:28Z

conflicts need to be resolved @raynay-r

cc @wanjunlei

raynay-r · 2024-07-01T07:29:15Z

Here is the output I get running make build on the main branch. You can also see in the output that I am using go 1.22.3. Is the go version maybe not compatible with the controller-gen version?

❯ go version
go version go1.22.3 linux/amd64
❯ make build
go get k8s.io/[email protected]
go mod download
/home/rene/Projects/fluent-operator/main/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa054af]

goroutine 354 [running]:
go/types.(*Checker).handleBailout(0xc0004a5400, 0xc001b35d40)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/check.go:367 +0x88
panic({0xbbf300?, 0x12ab8a0?})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/runtime/panic.go:770 +0x132
go/types.(*StdSizes).Sizeof(0x0, {0xdb6b18, 0x12b4040})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/sizes.go:228 +0x30f
go/types.(*Config).sizeof(...)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/sizes.go:333
go/types.representableConst.func1({0xdb6b18?, 0x12b4040?})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/const.go:76 +0x9e
go/types.representableConst({0xdbcdf0, 0x1280200}, 0xc0004a5400, 0x12b4040, 0x0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/const.go:92 +0x192
go/types.(*Checker).arrayLength(0xc0004a5400, {0xdbb140, 0xc001844b60?})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:510 +0x2d3
go/types.(*Checker).typInternal(0xc0004a5400, {0xdb9760, 0xc001ccbe60}, 0x0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:299 +0x49d
go/types.(*Checker).definedType(0xc0004a5400, {0xdb9760, 0xc001ccbe60}, 0xc001b35328?)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:180 +0x37
go/types.(*Checker).varType(0xc0004a5400, {0xdb9760, 0xc001ccbe60})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:145 +0x25
go/types.(*Checker).structType(0xc0004a5400, 0xc001d056b0, 0xc001d056b0?)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/struct.go:113 +0x19f
go/types.(*Checker).typInternal(0xc0004a5400, {0xdb96d0, 0xc001cce408}, 0xc0012c3db0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:316 +0x1345
go/types.(*Checker).definedType(0xc0004a5400, {0xdb96d0, 0xc001cce408}, 0xc8815d?)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:180 +0x37
go/types.(*Checker).typeDecl(0xc0004a5400, 0xc0012c3db0, 0xc001cd6580, 0x0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/decl.go:615 +0x44d
go/types.(*Checker).objDecl(0xc0004a5400, {0xdc23e0, 0xc0012c3db0}, 0x0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/decl.go:197 +0xa7f
go/types.(*Checker).packageObjects(0xc0004a5400)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/resolver.go:681 +0x425
go/types.(*Checker).checkFiles(0xc0004a5400, {0xc001527a40, 0x3, 0x3})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/check.go:408 +0x1a5
go/types.(*Checker).Files(...)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/check.go:372
sigs.k8s.io/controller-tools/pkg/loader.(*loader).typeCheck(0xc000363080, 0xc0007e65c0)
        /home/rene/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/loader/loader.go:286 +0x36a
sigs.k8s.io/controller-tools/pkg/loader.(*Package).NeedTypesInfo(0xc0007e65c0)
        /home/rene/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/loader/loader.go:99 +0x39
sigs.k8s.io/controller-tools/pkg/loader.(*TypeChecker).check(0xc0008c1bc0, 0xc0007e65c0)
        /home/rene/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/loader/refs.go:268 +0x2b7
sigs.k8s.io/controller-tools/pkg/loader.(*TypeChecker).check.func1(0x4f?)
        /home/rene/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/loader/refs.go:262 +0x53
created by sigs.k8s.io/controller-tools/pkg/loader.(*TypeChecker).check in goroutine 149
        /home/rene/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/loader/refs.go:260 +0x1c5
make: *** [Makefile:56: generate] Error 2

cw-Guo · 2024-07-01T20:29:30Z

Here is the output I get running make build on the main branch. You can also see in the output that I am using go 1.22.3. Is the go version maybe not compatible with the controller-gen version?

it's possible. Mine is 1.22.0, and it works well.
go version go1.22.0 darwin/arm64

Makefile

Signed-off-by: rene <[email protected]>

cw-Guo · 2024-07-04T23:18:24Z

apis/fluentbit/v1alpha2/plugins/output/loki_types.go

@@ -19,11 +19,16 @@ type Loki struct {
 	// +kubebuilder:validation:Minimum:=1
 	// +kubebuilder:validation:Maximum:=65535
 	Port *int32 `json:"port,omitempty"`
+	// Specify a custom HTTP URI. It must start with forward slash.
+	Uri string `json:"uri,omitempty"`


nit: is it better to use *string along with omitempty here to make it consistent with other fields?

benjaminhuo reviewed Jul 2, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

add bearer token parameter to fluent-bit (cluster) output

bb019e1

Signed-off-by: rene <[email protected]>

raynay-r force-pushed the add-bearer-token-auth-for-loki branch from 3b6fb01 to e11c855 Compare July 2, 2024 09:37

raynay-r added 2 commits July 2, 2024 12:17

add bearer token file parameter to fluentd (cluster) output

5e84ba0

Signed-off-by: rene <[email protected]>

add bearer token support in the helm chart for fluentbit loki output

f11d4c1

Signed-off-by: rene <[email protected]>

raynay-r force-pushed the add-bearer-token-auth-for-loki branch from e11c855 to f11d4c1 Compare July 2, 2024 10:17

cw-Guo reviewed Jul 4, 2024

View reviewed changes

benjaminhuo approved these changes Jul 8, 2024

View reviewed changes

benjaminhuo merged commit f39b24f into fluent:master Jul 8, 2024
9 of 10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bearer token auth for loki #1224

Add bearer token auth for loki #1224

raynay-r commented Jun 28, 2024

raynay-r commented Jun 28, 2024

cw-Guo commented Jul 1, 2024

benjaminhuo commented Jul 1, 2024

raynay-r commented Jul 1, 2024 •

edited

Loading

cw-Guo commented Jul 1, 2024

cw-Guo Jul 4, 2024 •

edited

Loading

Add bearer token auth for loki #1224

Add bearer token auth for loki #1224

Conversation

raynay-r commented Jun 28, 2024

What this PR does / why we need it:

Which issue(s) this PR fixes:

Does this PR introduced a user-facing change?

Additional documentation, usage docs, etc.:

raynay-r commented Jun 28, 2024

cw-Guo commented Jul 1, 2024

benjaminhuo commented Jul 1, 2024

raynay-r commented Jul 1, 2024 • edited Loading

cw-Guo commented Jul 1, 2024

cw-Guo Jul 4, 2024 • edited Loading

Choose a reason for hiding this comment

raynay-r commented Jul 1, 2024 •

edited

Loading

cw-Guo Jul 4, 2024 •

edited

Loading