Skip to content

Commit

Permalink
Merge pull request #1594 from csabatuz-chess/feature/filter-useragent
Browse files Browse the repository at this point in the history 
feat: add support for useragent Fluent filter
  • Loading branch information
@OverOrion
OverOrion authored Dec 5, 2023
2 parents a65274d + fcb9208 commit 7c2fa3e
Show file tree
Hide file tree
Showing 14 changed files with 377 additions and 0 deletions.
22 changes: 22 additions & 0 deletions charts/logging-operator/crds/logging.banzaicloud.io_clusterflows.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -965,6 +965,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down Expand Up @@ -1992,6 +2003,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down
22 changes: 22 additions & 0 deletions charts/logging-operator/crds/logging.banzaicloud.io_flows.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -965,6 +965,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down Expand Up @@ -1988,6 +1999,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down
22 changes: 22 additions & 0 deletions charts/logging-operator/crds/logging.banzaicloud.io_loggings.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1011,6 +1011,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down Expand Up @@ -7032,6 +7043,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
loggingRef:
Expand Down
22 changes: 22 additions & 0 deletions config/crd/bases/logging.banzaicloud.io_clusterflows.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -965,6 +965,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down Expand Up @@ -1992,6 +2003,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down
22 changes: 22 additions & 0 deletions config/crd/bases/logging.banzaicloud.io_flows.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -965,6 +965,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down Expand Up @@ -1988,6 +1999,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down
22 changes: 22 additions & 0 deletions config/crd/bases/logging.banzaicloud.io_loggings.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1011,6 +1011,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
flowLabel:
Expand Down Expand Up @@ -7032,6 +7043,17 @@ spec:
group_warning_delay_s:
type: integer
type: object
useragent:
properties:
delete_key:
type: boolean
flatten:
type: boolean
key_name:
type: string
out_key:
type: string
type: object
type: object
type: array
loggingRef:
Expand Down
3 changes: 3 additions & 0 deletions docs/configuration/crds/v1beta1/flow_types.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,9 @@ Filter definition for FlowSpec
### throttle (*filter.Throttle, optional) {#filter-throttle}


### useragent (*filter.UserAgent, optional) {#filter-useragent}



## FlowStatus

Expand Down
1 change: 1 addition & 0 deletions docs/configuration/plugins/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ For more information please click on the plugin name
| **[SumoLogic](filters/sumologic/)** | filters | Sumo Logic collection solution for Kubernetes | GA | [2.3.1](https://github.com/SumoLogic/sumologic-kubernetes-collection) |
| **[Tag Normaliser](filters/tagnormaliser/)** | filters | Re-tag based on log metadata | GA | [0.1.1](https://github.com/kube-logging/fluent-plugin-tag-normaliser) |
| **[Throttle](filters/throttle/)** | filters | A sentry plugin to throttle logs. Logs are grouped by a configurable key. When a group exceeds a configuration rate, logs are dropped for this group. | GA | [0.0.5](https://github.com/rubrikinc/fluent-plugin-throttle) |
| **[UserAgent](filters/useragent/)** | filters | Fluentd UserAgent filter | GA | [1.2.0](https://github.com/bungoume/fluent-plugin-ua-parser) |
| **[Amazon Elasticsearch](outputs/aws_elasticsearch/)** | outputs | Fluent plugin for Amazon Elasticsearch | Testing | [2.4.1](https://github.com/atomita/fluent-plugin-aws-elasticsearch-service) |
| **[Azure Storage](outputs/azurestore/)** | outputs | Store logs in Azure Storage | GA | [0.2.1](https://github.com/microsoft/fluent-plugin-azure-storage-append-blob) |
| **[Buffer](outputs/buffer/)** | outputs | Fluentd event buffer | GA | [mode info](https://docs.fluentd.org/configuration/buffer-section) |
Expand Down
72 changes: 72 additions & 0 deletions docs/configuration/plugins/filters/useragent.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
title: User Agent
weight: 200
generated_file: true
---

# Fluentd UserAgent filter
## Overview
Fluentd Filter plugin to parse user-agent
More information at https://github.com/bungoume/fluent-plugin-ua-parser

## Configuration
## UserAgent

### delete_key (bool, optional) {#useragent-delete_key}

Delete input key

Default: false

### flatten (bool, optional) {#useragent-flatten}

Join hashed data by '_'

Default: false

### key_name (string, optional) {#useragent-key_name}

Target key name

Default: user_agent

### out_key (string, optional) {#useragent-out_key}

Output prefix key name

Default: ua


## Example `UserAgent` filter configurations

{{< highlight yaml >}}
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: demo-flow
spec:
filters:
- useragent:
key_name: my_agent
delete_key: true
out_key: ua_fields
flatten: true
selectors: {}
localOutputRefs:
- demo-output
{{</ highlight >}}

## Fluentd Config Result
{{< highlight xml >}}
<filter **>
@type ua_parser
@id test_useragent
key_name my_agent
delete_key true
out_key ua_fields
flatten true
</filter>
{{</ highlight >}}


---
1 change: 1 addition & 0 deletions pkg/sdk/logging/api/v1beta1/flow_types.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ type Filter struct {
RecordTransformer *filter.RecordTransformer `json:"record_transformer,omitempty"`
RecordModifier *filter.RecordModifier `json:"record_modifier,omitempty"`
GeoIP *filter.GeoIP `json:"geoip,omitempty"`
UserAgent *filter.UserAgent `json:"useragent,omitempty"`
Concat *filter.Concat `json:"concat,omitempty"`
DetectExceptions *filter.DetectExceptions `json:"detectExceptions,omitempty"`
Grep *filter.GrepConfig `json:"grep,omitempty"`
Expand Down
5 changes: 5 additions & 0 deletions pkg/sdk/logging/api/v1beta1/zz_generated.deepcopy.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 7c2fa3e

Please sign in to comment.