logstash output error with json filter

[nosun]

Issue: I use logstash transfer data into mongodb, If I use filter json section to format field, I got error as below, but if I do not use filter section , mongodb can receive the data.

[2018-01-19T10:01:30,998][WARN ][logstash.outputs.mongodb ] Failed to send event to MongoDB {:event=>#<LogStash::Event:0x52e4bfb6>, :exception=>#<NoMethodError: undefined method `to_json' for nil:NilClass
Did you mean?  to_bson>, 

:backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-mongodb-3.1.3/lib/logstash/outputs/mongodb.rb:81:in `receive'", 

"/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:92:in `block in multi_receive'", "org/jruby/RubyArray.java:1734:in `each'", 

"/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:92:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/legacy.rb:22:in `multi_receive'", 

"/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:50:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:487:in `block in output_batch'", 

"org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:486:in `output_batch'", 

"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:438:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:393:in `block in start_workers'"]}

• Version: logstash-output-mongodb-3.1.3
• logstash version: 6.1
• Operating System: ubuntu 16.04
• Config File (if you have sensitive info, please remove it):

input {
  beats {
    port => 5044
    host => "0.0.0.0"
  }
}
filter {
  json {
     source => "message"
     remove_field => ["message"]
     add_field => {
      "timestamp" => "%{@timestamp}"
     }
  }
}
output {
   stdout { codec => json_lines }
   mongodb {
     uri => "mongodb://localhost:27017"
     database => "test"
     collection => "col1"
   }
}

• Sample Data:
  {'channelId': 'ios', 'BusinessRunTime': '2018-01-19 10:04:14.654'}
  {'channelId': 'ios', 'BusinessRunTime': '2018-01-19 10:04:24.663'}

• Steps to Reproduce:
  I use filebeat read a file with sample data, then transfer to logstash ,then transfer to mongo

[higee]

+1
@nosun Same here.
One thing I found is that I don't get that error once I remove remove_field inside json filter.
But I haven't yet found how to make this work with remove_field.
Did you figure it out?

[colinsurprenant]

seems related to #49 and #57