Fluentd - AWS managed opensearch service logging issue

[ekambaramv]

I need configure FluentD to send logs from my AWS-EKS nodes to AWS managed OpenSearch dashboard.

FluentD as DaemonSet installed on each of my AWS-EKS node.

Environment

- Fluentd version: 1.16.5
- TD Agent version:
- Fluent Package version:
- Docker image (tag): fluent/fluentd-kubernetes-daemonset:v1-debian-opensearch
- Operating system: amazon Linux 2 (EKS-Node)
- Kernel version:6.1.79-99.164.amzn2023.x86_64
- AWS EKS Kubernetes version: 1.29

What is a problem?

Problem:-
FluentD unable to communicate with Opensearch.

Describe the configuration of Fluentd

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-system
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
rules:
- apiGroups: [""]
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-system
---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      serviceAccount: fluentd
      serviceAccountName: fluentd
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-opensearch
        env:
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name:  FLUENT_OPENSEARCH_HOST
            value: "vpc-ms-opensearch-xxxx.on.aws"
          - name:  FLUENT_OPENSEARCH_PORT
            value: "443"
          - name: FLUENT_OPENSEARCH_SCHEME
            value: "https"
          - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
            value: "cri"
          - name: FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMAT
            value: "%Y-%m-%dT%H:%M:%S.%N%:z"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: dockercontainerlogdirectory
          mountPath: /var/log/pods
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: dockercontainerlogdirectory
        hostPath:
          path: /var/log/pods

Describe the logs of Fluentd

$ kubectl logs fluentd-2l9ck -n kube-system
2024-05-06 09:42:03 +0000 [info]: starting fluentd-1.16.5 pid=7 ruby="3.2.3"
2024-05-06 09:42:03 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/3.2.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
2024-05-06 09:42:03 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2024-05-06 09:42:03 +0000 [info]: adding match in @FLUENT_LOG pattern="fluent.**" type="null"
2024-05-06 09:42:03 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2024-05-06 09:42:04 +0000 [info]: adding match pattern="**" type="opensearch"
2024-05-06 09:42:06 +0000 [warn]: #0 [out_os] Could not communicate to OpenSearch, resetting connection and trying again. No such file or directory @ rb_sysopen -  (Errno::ENOENT)
2024-05-06 09:42:06 +0000 [warn]: #0 [out_os] Remaining retry: 14. Retry to communicate after 2 second(s).
2024-05-06 09:42:10 +0000 [warn]: #0 [out_os] Could not communicate to OpenSearch, resetting connection and trying again. No such file or directory @ rb_sysopen -  (Errno::ENOENT)
2024-05-06 09:42:10 +0000 [warn]: #0 [out_os] Remaining retry: 13. Retry to communicate after 4 second(s).
2024-05-06 09:42:18 +0000 [warn]: #0 [out_os] Could not communicate to OpenSearch, resetting connection and trying again. No such file or directory @ rb_sysopen -  (Errno::ENOENT)
2024-05-06 09:42:18 +0000 [warn]: #0 [out_os] Remaining retry: 12. Retry to communicate after 8 second(s).
2024-05-06 09:42:34 +0000 [warn]: #0 [out_os] Could not communicate to OpenSearch, resetting connection and trying again. No such file or directory @ rb_sysopen -  (Errno::ENOENT)
2024-05-06 09:42:34 +0000 [warn]: #0 [out_os] Remaining retry: 11. Retry to communicate after 16 second(s).
2024-05-06 09:43:06 +0000 [warn]: #0 [out_os] Could not communicate to OpenSearch, resetting connection and trying again. No such file or directory @ rb_sysopen -  (Errno::ENOENT)
2024-05-06 09:43:06 +0000 [warn]: #0 [out_os] Remaining retry: 10. Retry to communicate after 32 second(s).
2024-05-06 09:44:11 +0000 [warn]: #0 [out_os] Could not communicate to OpenSearch, resetting connection and trying again. No such file or directory @ rb_sysopen -  (Errno::ENOENT)
2024-05-06 09:44:11 +0000 [warn]: #0 [out_os] Remaining retry: 9. Retry to communicate after 64 second(s).

# fluentd -c /fluentd/etc/kubernetes.conf -d -vv
/var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/basic_parser.rb:92:in `parse_error!': unmatched end tag at kubernetes.conf line 48,9 (Fluent::ConfigParseError)
 47: <% end %>
 48: </source>
     ---------^
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:78:in `parse_element'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:95:in `parse_element'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:95:in `parse_element'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:95:in `parse_element'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:95:in `parse_element'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:43:in `parse!'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/config/v1_parser.rb:33:in `parse'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/config.rb:39:in `parse'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:766:in `read_config'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/supervisor.rb:477:in `run_supervisor'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/command/fluentd.rb:310:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /var/lib/gems/2.3.0/gems/fluentd-1.2.6/bin/fluentd:8:in `<top (required)>'
        from /usr/local/bin/fluentd:22:in `load'
        from /usr/local/bin/fluentd:22:in `<main>'

Please let me know what i am missing as part of my setup/configurations? Any help on this would be highly appreciated!