Implement HTTP feature tests

[edsiper]

Since in Fluent Bit, we use different mechanisms to handle HTTP requests for versions HTTP/1.x and HTTP/2, different plugins that use this interface (HTTP server) behind the scenes might have different ways to deal with requests, and recently we found some issues when mixing different configuration options.

In this issue, we will focus on testing the following input plugins only:

• in_http
• in_splunk
• in_elasticsearch

what needs to be tested

The latest version from Fluent Bit in GIT master needs to be tested against the following matrix of configuration options:

• http2: on
• http2: off
• net.keepalive: on
• net.keepalive: off
• tls: on
• tls: off

This is one example configured for Splunk on http2 on, keepalive on, tls off and with a HTTP/1.1 request (it should work), e.g:

pipeline:
  inputs:
    - name: splunk
      port: 9883
      host: 0.0.0.0
      http2: on
      net.keepalive: on
      tls: off
  outputs:
    - name: stdout
      match: '*'

the following Python3 code can be used to test the Splunk server:

import http.client
import json

# Define the server and port
server = 'localhost'
port = 9883

# Create a connection
conn = http.client.HTTPConnection(server, port)

# Define headers
headers = {
    'Connection': 'keepalive',
    'Authorization': 'Splunk secret-token',
    'Content-Type': 'application/json'
}

# Define the payload
payload = {
    "User": "Admin",
    "password": "my_secret_password",
    "Event": "Some text in the event"
}

# Convert payload to JSON
json_payload = json.dumps(payload)

# Define the number of requests
num_requests = 3

# Perform the requests
for i in range(num_requests):
    conn.request("POST", "/services/collector", body=json_payload, headers=headers)
    response = conn.getresponse()
    print(f"Request {i+1} status: {response.status}, reason: {response.reason}")
    response_data = response.read().decode()
    print(f"Response {i+1} data: {response_data}")

# Close the connection
conn.close()

The code above does 3 requests within a keepalive connection, the expectation is to have the following outputs:

Fluent Bit output (stdout)

[0] splunk.0: [[1720042449.585015000, {"hec_token"=>"Splunk secret-token"}], {"User"=>"Admin", "password"=>"my_secret_password", "Event"=>"Some text in the event"}]
[1] splunk.0: [[1720042449.585317000, {"hec_token"=>"Splunk secret-token"}], {"User"=>"Admin", "password"=>"my_secret_password", "Event"=>"Some text in the event"}]
[2] splunk.0: [[1720042449.585552000, {"hec_token"=>"Splunk secret-token"}], {"User"=>"Admin", "password"=>"my_secret_password", "Event"=>"Some text in the event"}]

Python code output

Request 1 status: 200, reason: OK
Response 1 data: {"text":"Success","code":0}
Request 2 status: 200, reason: OK
Response 2 data: {"text":"Success","code":0}
Request 3 status: 200, reason: OK
Response 3 data: {"text":"Success","code":0}

Notes when testing with TLS enabled

TLS server side needs to use some certificates, we can use the ones available for testing inside Fluent Bit code repo, just doing a copy is enough:

• certificate.pem
• private_key.pem

Here is an example of Splunk input plugin with TLS enabled:

pipeline:
  inputs:
    - name: splunk
      port: 9883
      host: 0.0.0.0
      http2: off
      net.keepalive: on
      tls: On
      tls.crt_file: ../tests/runtime/data/tls/certificate.pem
      tls.key_file: ../tests/runtime/data/tls/private_key.pem
  outputs:
    - name: stdout
      match: '*'

[edsiper]

cc: @odonata

[odonata]

This issue is resolved.