Access log to Elasticsearch stop working

[underscoredje]

Hi!

I made a fresh install of a new vulture cluster on freebsd12.1 and connected a ES repo for logs (as I I did for my older Vulture cluster).

On this new cluster:

• Pf logs are send to ES, but access logs are not. (On ES vulture_pf-* index is created, but vulture_access-* is not).
• I did some checks /usr/local/etc/rsyslog.d/rsyslogd.conf file and did some tests with the part concerning my access log file:

input(type="imfile" File="/var/log/Vulture/worker/vulture-test.test.com--access.log" Tag="5f8d5064658ec17905c6c1b0")


if $syslogtag contains '5f8d5064658ec17905c6c1b0' then {
  action(type="mmjsonparse")
  $template srchidx_5f8d5064658ec17905c6c1b0,"vulture_access-%timegenerated:::date-year%-%timegenerated:::date-month%-%timegenerated:::date-day%"

  

  ### Send logs to Elastic ###
  if $parsesuccess == "OK" then {
      if $!score == "-" then {
          set $!score = 0;
      } else {
          set $!score = cnum($!score);
      }
      if $!threshold == "-" then {
          set $!threshold = 0;
      } else {
          set $!threshold = cnum($!threshold);
      }

      action(type="omelasticsearch"
           server=["http://es.test.lan:9200"]
           template="tpl_elastic"
           searchIndex="srchidx_5f8d5064658ec17905c6c1b0"
           dynSearchIndex="on"
           searchType="vulture_access"
           
           bulkmode="on"
           maxbytes="100m"
           queue.type="linkedlist"
           queue.size="5000"
           queue.dequeuebatchsize="300"
           action.resumeretrycount="-1")
  }
  stop
}

If I comment if $parsesuccess == "OK" then { line, and the correponding } line, vulture-access-* is created in ES. (but with wrong parsing).

So, i guess there is a problem with the parsing made by mmjsonparse.
I've tested with rsyslogd 8.1911.0 and 8.2008.0.

On older clusters (where it is working), rsyslogd 8.40.0 is used.

Any idea for correcting this ?

[underscoredje]

After more investigations, it seems the the problem is in the access_log formatting:

@cee:{"app_name":"test","src_ip":"****","user":"-","time":"2020-10-19T14:02:09+0000","http_method":"GET","requested_uri":"/,"http_code":304,"incoming_protocol":"HTTP/1.1","referer":"https://test.test.com/","user_agent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0","size":0,"bytes_received":702,"bytes_sent":177,"time_elapsed":2094,"country":"FR","city":"-","lat":"-","lon":"-","reputation":"-,-,-,-,-","owasp_top10":"-","reasons":"[]","threshold":"18","score":"0"}"authentication":"-","authentication_result":"-"}

Note the appending "authentication":"-","authentication_result":"-"}

[underscoredje]

OK, found the root cause and corrected it by modifying /home/vlt-gui/vulture/gui/models/modlog_settings.py.
I will submit a patch.

[underscoredje]

PR submitted #12

[b4b857f6ee]

@underscoredje Brooo help me too :D.
I got the vulture_pf working on the ELK. i have to modify the file like your first post?

[b4b857f6ee]

Oh ok this is guess :

[underscoredje]

Yes!
Hope it is working for you now.

[b4b857f6ee]

@underscoredje Yes, i'm starting the Vulture Grafana dashboard to night.

[KGuillemot]

Hello @underscoredje,
PR #12 has been merged, can you confirm me your problem is solved please ?