Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bearer token auth for loki #1224

Merged
merged 3 commits into from
Jul 8, 2024
Merged

Add bearer token auth for loki #1224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bb019e1
add bearer token parameter to fluent-bit (cluster) output
raynay-r Jun 28, 2024
5e84ba0
add bearer token file parameter to fluentd (cluster) output
raynay-r Jun 28, 2024
f11d4c1
add bearer token support in the helm chart for fluentbit loki output
raynay-r Jul 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions apis/fluentbit/v1alpha2/plugins/output/loki_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,16 @@ type Loki struct {
// +kubebuilder:validation:Minimum:=1
// +kubebuilder:validation:Maximum:=65535
Port *int32 `json:"port,omitempty"`
// Specify a custom HTTP URI. It must start with forward slash.
Uri string `json:"uri,omitempty"`
Copy link
Collaborator

@cw-Guo cw-Guo Jul 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is it better to use *string along with omitempty here to make it consistent with other fields?

// Set HTTP basic authentication user name.
HTTPUser *plugins.Secret `json:"httpUser,omitempty"`
// Password for user defined in HTTP_User
// Set HTTP basic authentication password
HTTPPasswd *plugins.Secret `json:"httpPassword,omitempty"`
// Set bearer token authentication token value.
// Can be used as alterntative to HTTP basic authentication
BearerToken *plugins.Secret `json:"bearerToken,omitempty"`
// Tenant ID used by default to push logs to Loki.
// If omitted or empty it assumes Loki is running in single-tenant mode and no X-Scope-OrgID header is sent.
TenantID *plugins.Secret `json:"tenantID,omitempty"`
Expand Down Expand Up @@ -70,6 +75,9 @@ func (l *Loki) Params(sl plugins.SecretLoader) (*params.KVs, error) {
if l.Port != nil {
kvs.Insert("port", fmt.Sprint(*l.Port))
}
if l.Uri != "" {
kvs.Insert("uri", l.Uri)
}
if l.HTTPUser != nil {
u, err := sl.LoadSecret(*l.HTTPUser)
if err != nil {
Expand All @@ -84,6 +92,13 @@ func (l *Loki) Params(sl plugins.SecretLoader) (*params.KVs, error) {
}
kvs.Insert("http_passwd", pwd)
}
if l.BearerToken != nil {
bearerToken, err := sl.LoadSecret(*l.BearerToken)
if err != nil {
return nil, err
}
kvs.Insert("bearer_token", bearerToken)
}
if l.TenantID != nil {
id, err := sl.LoadSecret(*l.TenantID)
if err != nil {
Expand Down
5 changes: 5 additions & 0 deletions apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions apis/fluentd/v1alpha1/plugins/output/loki.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ type Loki struct {
// Password for user defined in HTTP_User
// Set HTTP basic authentication password
HTTPPasswd *plugins.Secret `json:"httpPassword,omitempty"`
// Set path to file with bearer authentication token
// Can be used as alterntative to HTTP basic authentication
BearerTokenFile *string `json:"bearerTokenFile,omitempty"`
// Tenant ID used by default to push logs to Loki.
// If omitted or empty it assumes Loki is running in single-tenant mode and no X-Scope-OrgID header is sent.
TenantID *plugins.Secret `json:"tenantID,omitempty"`
Expand Down
3 changes: 3 additions & 0 deletions apis/fluentd/v1alpha1/plugins/output/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -714,6 +714,9 @@ func (o *Output) lokiPlugin(parent *params.PluginStore, loader plugins.SecretLoa
}
parent.InsertPairs("password", passwd)
}
if o.Loki.BearerTokenFile != nil {
parent.InsertPairs("bearer_token_file", fmt.Sprint(*o.Loki.BearerTokenFile))
}
if o.Loki.TenantID != nil {
id, err := loader.LoadSecret(*o.Loki.TenantID)
if err != nil {
Expand Down
35 changes: 35 additions & 0 deletions charts/fluent-operator/charts/fluent-bit-crds/crds/fluentbit.fluent.io_clusteroutputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1979,6 +1979,37 @@ spec:
- "on"
- "off"
type: string
bearerToken:
description: |-
Set bearer token authentication token value.
Can be used as alterntative to HTTP basic authentication
properties:
valueFrom:
description: ValueSource defines how to find a value's key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's namespace
properties:
key:
description: The key of the secret to select from. Must
be a valid secret key.
type: string
name:
description: |-
Name of the referent.
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Add other useful fields. apiVersion, kind, uid?
type: string
optional:
description: Specify whether the Secret or its key
must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
dropSingleKey:
description: If set to true and after extracting labels only a
single key remains, the log line sent to Loki will be the value
Expand Down Expand Up @@ -2249,6 +2280,10 @@ spec:
description: Hostname to be used for TLS SNI extension
type: string
type: object
uri:
description: Specify a custom HTTP URI. It must start with forward
slash.
type: string
required:
- host
type: object
Expand Down
35 changes: 35 additions & 0 deletions charts/fluent-operator/charts/fluent-bit-crds/crds/fluentbit.fluent.io_outputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1979,6 +1979,37 @@ spec:
- "on"
- "off"
type: string
bearerToken:
description: |-
Set bearer token authentication token value.
Can be used as alterntative to HTTP basic authentication
properties:
valueFrom:
description: ValueSource defines how to find a value's key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's namespace
properties:
key:
description: The key of the secret to select from. Must
be a valid secret key.
type: string
name:
description: |-
Name of the referent.
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Add other useful fields. apiVersion, kind, uid?
type: string
optional:
description: Specify whether the Secret or its key
must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
dropSingleKey:
description: If set to true and after extracting labels only a
single key remains, the log line sent to Loki will be the value
Expand Down Expand Up @@ -2249,6 +2280,10 @@ spec:
description: Hostname to be used for TLS SNI extension
type: string
type: object
uri:
description: Specify a custom HTTP URI. It must start with forward
slash.
type: string
required:
- host
type: object
Expand Down
5 changes: 5 additions & 0 deletions charts/fluent-operator/charts/fluentd-crds/crds/fluentd.fluent.io_clusteroutputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1914,6 +1914,11 @@ spec:
loki:
description: out_loki plugin
properties:
bearerTokenFile:
description: |-
Set path to file with bearer authentication token
Can be used as alterntative to HTTP basic authentication
type: string
dropSingleKey:
description: If a record only has 1 key, then just set the
log line to the value and discard the key.
Expand Down
5 changes: 5 additions & 0 deletions charts/fluent-operator/charts/fluentd-crds/crds/fluentd.fluent.io_outputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1914,6 +1914,11 @@ spec:
loki:
description: out_loki plugin
properties:
bearerTokenFile:
description: |-
Set path to file with bearer authentication token
Can be used as alterntative to HTTP basic authentication
type: string
dropSingleKey:
description: If a record only has 1 key, then just set the
log line to the value and discard the key.
Expand Down
18 changes: 16 additions & 2 deletions charts/fluent-operator/templates/fluentbit-output-loki.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,15 @@

{{ with .Values.fluentbit.output.loki -}}
{{/*
When http{User,Password} or tenantID is a string, make a secret for them
When http{User,Password}, bearerToken, or tenantID is a string, make a secret for them
When these keys are objects, they specify a secret to use generated elsewhere, assumed to exist in the k8s cluster
*/}}
{{ $userSecret := "loki-http-auth" -}}
{{ $passSecret := "loki-http-pass" -}}
{{ $bearerTokenSecret := "loki-bearer-token" -}}
{{ $tenantIDSecret := "loki-tenant-id" -}}

{{ range $k, $v := dict $userSecret .httpUser $passSecret .httpPassword $tenantIDSecret .tenantID -}}
{{ range $k, $v := dict $userSecret .httpUser $passSecret .httpPassword $tenantIDSecret .tenantID $bearerTokenSecret .bearerToken -}}
{{ if kindIs "string" $v -}}
---
apiVersion: v1
Expand Down Expand Up @@ -81,6 +82,19 @@ spec:
{{- end }}
{{- end }}

{{- if .bearerToken }}
bearerToken:
{{- if kindIs "string" .bearerToken }}
valueFrom:
secretKeyRef:
key: 'value'
name: {{ $bearerTokenSecret }}
optional: false
{{- else }}
{{ .bearerToken | toYaml | indent 6 }}
{{- end }}
{{- end }}

{{- if .tenantID }}
tenantID:
{{- if kindIs "string" .tenantID }}
Expand Down
11 changes: 10 additions & 1 deletion charts/fluent-operator/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ fluentbit:
# See https://docs.fluentbit.io/manual/pipeline/outputs/loki
loki:
# Switch for generation of fluentbit loki ClusterOutput (and loki basic auth http user and pass secrets if required)
enable: false # Bool
enable: false # Bool
host: 127.0.0.1 # String
port: 3100 # Int
# Either, give http{User,Password},tenantID string values specifying them directly
Expand All @@ -305,6 +305,15 @@ fluentbit:
# name: tenantsecret
# optional: true
#
# To use bearer token auth instead of http basic auth
#bearerToken: ey....
# or with existing secret
#bearerToken:
# valueFrom:
# secretKeyRef:
# key: value
# name: bearerTokenSecret
# optional: true
#labels: [] # String list of <name>=<value>
#labelKeys: [] # String list of <key>
#removeKeys: [] # String list of <key>
Expand Down
35 changes: 35 additions & 0 deletions config/crd/bases/fluentbit.fluent.io_clusteroutputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1979,6 +1979,37 @@ spec:
- "on"
- "off"
type: string
bearerToken:
description: |-
Set bearer token authentication token value.
Can be used as alterntative to HTTP basic authentication
properties:
valueFrom:
description: ValueSource defines how to find a value's key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's namespace
properties:
key:
description: The key of the secret to select from. Must
be a valid secret key.
type: string
name:
description: |-
Name of the referent.
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Add other useful fields. apiVersion, kind, uid?
type: string
optional:
description: Specify whether the Secret or its key
must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
dropSingleKey:
description: If set to true and after extracting labels only a
single key remains, the log line sent to Loki will be the value
Expand Down Expand Up @@ -2249,6 +2280,10 @@ spec:
description: Hostname to be used for TLS SNI extension
type: string
type: object
uri:
description: Specify a custom HTTP URI. It must start with forward
slash.
type: string
required:
- host
type: object
Expand Down
35 changes: 35 additions & 0 deletions config/crd/bases/fluentbit.fluent.io_outputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1979,6 +1979,37 @@ spec:
- "on"
- "off"
type: string
bearerToken:
description: |-
Set bearer token authentication token value.
Can be used as alterntative to HTTP basic authentication
properties:
valueFrom:
description: ValueSource defines how to find a value's key.
properties:
secretKeyRef:
description: Selects a key of a secret in the pod's namespace
properties:
key:
description: The key of the secret to select from. Must
be a valid secret key.
type: string
name:
description: |-
Name of the referent.
More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
TODO: Add other useful fields. apiVersion, kind, uid?
type: string
optional:
description: Specify whether the Secret or its key
must be defined
type: boolean
required:
- key
type: object
x-kubernetes-map-type: atomic
type: object
type: object
dropSingleKey:
description: If set to true and after extracting labels only a
single key remains, the log line sent to Loki will be the value
Expand Down Expand Up @@ -2249,6 +2280,10 @@ spec:
description: Hostname to be used for TLS SNI extension
type: string
type: object
uri:
description: Specify a custom HTTP URI. It must start with forward
slash.
type: string
required:
- host
type: object
Expand Down
5 changes: 5 additions & 0 deletions config/crd/bases/fluentd.fluent.io_clusteroutputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1914,6 +1914,11 @@ spec:
loki:
description: out_loki plugin
properties:
bearerTokenFile:
description: |-
Set path to file with bearer authentication token
Can be used as alterntative to HTTP basic authentication
type: string
dropSingleKey:
description: If a record only has 1 key, then just set the
log line to the value and discard the key.
Expand Down
5 changes: 5 additions & 0 deletions config/crd/bases/fluentd.fluent.io_outputs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1914,6 +1914,11 @@ spec:
loki:
description: out_loki plugin
properties:
bearerTokenFile:
description: |-
Set path to file with bearer authentication token
Can be used as alterntative to HTTP basic authentication
type: string
dropSingleKey:
description: If a record only has 1 key, then just set the
log line to the value and discard the key.
Expand Down
2 changes: 2 additions & 0 deletions docs/plugins/fluentbit/output/loki.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,10 @@ The loki output plugin, allows to ingest your records into a Loki service. <br /
| ----- | ----------- | ------ |
| host | Loki hostname or IP address. | string |
| port | Loki TCP port | *int32 |
| uri | Specify a custom HTTP URI. It must start with forward slash. | string |
| httpUser | Set HTTP basic authentication user name. | *[plugins.Secret](../secret.md) |
| httpPassword | Password for user defined in HTTP_User Set HTTP basic authentication password | *[plugins.Secret](../secret.md) |
| bearerToken | Set bearer token authentication token value. Can be used as alterntative to HTTP basic authentication | *[plugins.Secret](../secret.md) |
| tenantID | Tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no X-Scope-OrgID header is sent. | *[plugins.Secret](../secret.md) |
| labels | Stream labels for API request. It can be multiple comma separated of strings specifying key=value pairs. In addition to fixed parameters, it also allows to add custom record keys (similar to label_keys property). | []string |
| labelKeys | Optional list of record keys that will be placed as stream labels. This configuration property is for records key only. | []string |
Expand Down
1 change: 1 addition & 0 deletions docs/plugins/fluentd/output/loki.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ The loki output plugin, allows to ingest your records into a Loki service.
| url | Loki URL. | *string |
| httpUser | Set HTTP basic authentication user name. | *[plugins.Secret](../secret.md) |
| httpPassword | Password for user defined in HTTP_User Set HTTP basic authentication password | *[plugins.Secret](../secret.md) |
| bearerTokenFile | Set path to file with bearer authentication token Can be used as alterntative to HTTP basic authentication | *string |
| tenantID | Tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no X-Scope-OrgID header is sent. | *[plugins.Secret](../secret.md) |
| labels | Stream labels for API request. It can be multiple comma separated of strings specifying key=value pairs. In addition to fixed parameters, it also allows to add custom record keys (similar to label_keys property). | []string |
| labelKeys | Optional list of record keys that will be placed as stream labels. This configuration property is for records key only. | []string |
Expand Down
Loading
Loading