Fluentbit config parsing logic for isolated region compatibility

[whoix]

Issue #, if available:

Description of changes:

Isolated regions need the Cloudwatch logs endpoint specified in the linux fluent bit configmap in order to properly create log groups. The endpoints are different for isolated (ADC) regions and do not follow conventional formatting compared to commercial. I have refactored some logic to properly parse which region the addon is being applied to and appropriately apply the correct linux configmap. These changes already work in isolated regions and are in AWS code base.

###Testing
Images/addon components are already onboarded to an internal ImageReplicationService. This automatically syncs CW images lowside and transfers them up to all supported EKS regions. We have worked with the EKS team to ensure you guys are already onboarded.

I was able to confirm metrics, log groups, and that my metrics for Application Signals and GPU Container insights show up in Container Insights. I am not able to attach/illustrate specific screenshots or logs due to security implications.

Changes only affect ADC regions to incorporate the endpoint line to the configmap. Commercial is unaffected and i did test there too.

In commercial the configmap remains the same

[INPUT]
  Name                tail
  Tag                 application.*
  Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
  Path                /var/log/containers/*.log
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_container.db
  Mem_Buf_Limit       50MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Rotate_Wait         30
  storage.type        filesystem
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/fluent-bit*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_log.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/cloudwatch-agent*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_cwagent.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
  Name                kubernetes
  Match               application.*
  Kube_URL            https://kubernetes.default.svc:443
  Kube_Tag_Prefix     application.var.log.containers.
  Merge_Log           On
  Merge_Log_Key       log_processed
  K8S-Logging.Parser  On
  K8S-Logging.Exclude Off
  Labels              Off
  Annotations         Off
  Use_Kubelet         On
  Kubelet_Port        10250
  Buffer_Size         0

[OUTPUT]
  Name                cloudwatch_logs
  Match               application.*
  region              ${AWS_REGION}
  log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
  log_stream_prefix   ${HOST_NAME}-
  auto_create_group   true
  extra_user_agent    container-insights

while Isolated region now have the new added endpoint param

[INPUT]
  Name                tail
  Tag                 application.*
  Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
  Path                /var/log/containers/*.log
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_container.db
  Mem_Buf_Limit       50MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Rotate_Wait         30
  storage.type        filesystem
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/fluent-bit*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_log.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[INPUT]
  Name                tail
  Tag                 application.*
  Path                /var/log/containers/cloudwatch-agent*
  multiline.parser    docker, cri
  DB                  /var/fluent-bit/state/flb_cwagent.db
  Mem_Buf_Limit       5MB
  Skip_Long_Lines     On
  Refresh_Interval    10
  Read_from_Head      ${READ_FROM_HEAD}

[FILTER]
  Name                kubernetes
  Match               application.*
  Kube_URL            https://kubernetes.default.svc:443
  Kube_Tag_Prefix     application.var.log.containers.
  Merge_Log           On
  Merge_Log_Key       log_processed
  K8S-Logging.Parser  On
  K8S-Logging.Exclude Off
  Labels              Off
  Annotations         Off
  Use_Kubelet         On
  Kubelet_Port        10250
  Buffer_Size         0

[OUTPUT]
  Name                cloudwatch_logs
  Match               application.*
  region              ${AWS_REGION}
  log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
  log_stream_prefix   ${HOST_NAME}-
  auto_create_group   true
  endpoint             logs.${AWS_REGION}.c2s.ic.gov
  extra_user_agent    container-insights

[jefchien]

If the only difference is the endpoint, could we figure out a way to template it out instead of duplicating each of the configs?

[whoix]

The problem is that you would need to refactor your fluent-bit-configmap in your linux helm chart to parse only for the OUTPUTS section and for ADC regions which then gets convoluted. Its about the same refactoring as I have done already in the values currently. The endpoint only needs to be specified in the OUTPUTS section, other wise it can actually cause helm chart formatting errors -> causes the pod to be instable.

Open to suggestions but this was the safest way i could think that doesn't impact your helm charts too much.

[jefchien]

Ideally, we would fix this in fluent bit so the right endpoint would be used without having to override it https://github.com/fluent/fluent-bit/blob/master/src/aws/flb_aws_util.c#L75, but until that's done, I'm fine with doing it this way.

[jefchien]

nit: Could use hasPrefix instead https://helm.sh/docs/chart_template_guide/function_list/#hasprefix-and-hassuffix

[whoix]

Ack. updating.