Honeytail nginx parser doesn't recognize log_format defined in included config.

[es1o]

Versions

• Go:
• Honeytail: 1.8.2

Steps to reproduce

1. Have nginx.conf file without log definition
2. Add include /etc/nginx/conf.d/*.conf
3. Add file /etc/nginx/conf.d/app.conf with log definition, e.g. log_format combined_apm...
4. Run honeytail with command: honeytail --parser=nginx --nginx.format=combined_apm --dataset=my-dataset --nginx.conf=/etc/nginx/nginx.conf --file=/var/log/nginx.log --status_interval=1 --add_field service.name=my-service

In this case honeytail fails with error:

FATA[0000] Error initializing nginx parser module: `log_format combined_apm` not found in given config

Additional context

I tried use --nginx.conf=/etc/nginx/conf.d/app.conf but I have the same error.

[kentquirk]

Hi, @es1o. If I understand what you're doing, I think you're:

• Specifying an include statement in nginx.conf that includes conf.d/app.conf
• Expecting honeytail to read app.conf to extract the format for the log.

Honeytail doesn't do that; it doesn't contain a general-purpose conf parser, so it doesn't know how to do include.

It sounds to me like you tried to do what I was about to suggest instead, which is specify the full path to app.conf. Since that didn't work, is there any chance you could share a sanitized app.conf file so we can try to reproduce?

[es1o]

Hi, @kentquirk I found an issue. I have log with newline after log name:

  log_format combinedapm 
      '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '

It is a valid nginx.conf but it is not parsed correctly by honeytail.

[kentquirk]

FYI -- investigating this issue uncovered a chain of old dependencies for nginx config parsing that needs some work to unravel. When that's done, we may be able to support more advanced log format statements in some future release.

For now, the parser isn't going to recognize line concatenation like this, and you'll need to put the entire log format on one line, or use a regex parser instead.