Filter Multiline

[ibigpapa]

"Moving https://logstash.jira.com/browse/LOGSTASH-631 to Github

Rails production.log has two empty lines in between each event. My multiline filter with pattern "\n\n" is not matching and I know for sure I have two newlines.
Here is the filter I tried which isn't working with a typical rails production.log:
multiline
{ type => "rails_prod_log" pattern => "\n\n" negate => "true" what => "previous" }
The following works but isn't quite what I want:
multiline
{ type => "rails_prod_log" pattern => "^$" negate => "true" what => "previous" }
The reason:
Sometimes an event/stacktrace has a broken single empty line with one newline character in it, in between the stacktrace. I wouldn't want to split that stacktrace into two separate events so I need to match on two newlines (ie "\n\n")
I'd appreciate a work-around for now until version 1.1.2 comes out.
Thanks!
Navid"

Confirmed still occuring in 1.5

[ibigpapa]

i create a test file to use with i'm trying to make the whole file 1 event for testing purposes

#: is not apart of the file just using to show when i skip a line

[FILE]
1: first line
2: second line
3: third line
4:
5: fifth line skipped one
6:
7:
8: eighth line skipped 2
9:
[end file]

[wiibaa]

@ibigpapa that's an interesting use case. I don't think it is achievable in a single multiline filter because the filter receive event line by line therefore \n\n cannot be matched for sure.

Here is a proposal I worked on with the stdin input, it might require some adaptation but I hope we can reach your use case. I needed \r* here because stdin input add them on windows.

filter {
    multiline {
      what => next
      pattern => "^\r*$"
      negate => true
    }

    multiline {
      what => previous
      pattern => "(?m)\A\r*\Z"
      negate => true
    }
}

[codearoo]

Hi.. this may not be exactly related but I'm seeing what I consider a bug in multiline.

For some reason, everyone is always excited to talk about using multiline.. what => previous but that to me is not ideal, and it has the drawback of causing the system to get stalled and not receive a log entry after that which might have just 1 line. Plus what if the data is not beginning with a space?

Anyway, what makes much more sense is to identify what the BEGINNING of a log entry looks like and use what => "next"
Unfortunately, that feature does not seem to work.

What ends up happening if I have a log entry that spans more than 2 lines, is that it correctly puts lines 1 and 2 together, but somehow 3rd line ends up in its own message.

It really should work that simply, and not care if grok or whatever follows or not.. which I'm not seeing making a difference anyway. So I believe this is a bug.

[codearoo]

Using multiline in filter because I get various stuff other servers over TCP.. so here is my section of the config:

 multiline {
     pattern => "^\[.*\]"
     what => "next"
     stream_identity => "%{host}.%{FileName}"
 }

[jordansissel]

For Logstash 1.5.0, we've moved all plugins to individual repositories, so I have moved this issue to logstash-plugins/logstash-filter-multiline#13. Let's continue the discussion there! :)