fix: workaround for some worker not autherized.

[kevinwang2]

username and password are correct. but cannot login when first
insert occur. This patch avoid continue to use the not autherized
connection and try to re-login.

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run; then say 'jenkins, test it'.

[kevinwang2]

I have already sign the CLA. Please help me check it out. Thanks!

[guyboertje]

@kevinwang2 - I believe this Mongo JIRA ticket 1049 describes the problem you are having.

In the discussion of the JIRA ticket, it seems that there are two sources of not authorized or unauthorized

1. when the user is genuinely not authorized for the insert operation.
2. when the server is very busy and it cannot get the global lock to check if the user is authorized

Your patch works for 2, but for 1 will retry forever (or until the mongodb user privilege is changed).

Consider using a check in the register method to test for the insert privilege for the current user.
using maybe admin.system.users, and raising an error if missing.
Perhaps you can ask the ruby mongo client people how to query for the privileges of the current user.

I think it would better if LS did not start at all if the current user is not privileged enough to insert on the configured collection and db.

[kevinwang2]

@guyboertje - I agree the register method are a more graceful place to place some fix code. But I can't do that.

One reason is Logstash cannot access admin.system.users collection because It require some special role to touch it. So this advice cannot be implement. Sorry.

I think this patch can only handle situation 1. But for situation 2 the processing are (at least) not worse than nothing.

In my case (situation 1 with 10 workers), the logstash has 2% chance to create a thread with not authorized connection. Then generating a lot of error logs both logstash and mongodb side (2GB log each side and each day). It really horrible for our ops team.

Btw, Sometime logstash will doing some auto-reconnect. So, even no error in startup. Bad connections may still appeared in tomorrow or midnight.