Too many config files

[estezz]

I have 6 topics each with 3 partitions. that makes 18 config files and processes. It there a way to reduce this?

[reachkrishnaraj]

@estezz : Thanks, Yes, thats a good point which I am already thinking to implement. Should not take more than 1 hr of time. I will work on it and let you know. There are 2 ways to do this.1

1. Keep the consumer job single threaded job which will be tied to each partition. And have 1 common property file for all the consumer instances(without the partition parameter) and pass the topic as command line parameter while starting the job.

2. Multiple thread the consumer job, 1 thread for each topic. And all the topics can be put in a comma separated single property.

[estezz]

I think #2 sounds best. It will make monitoring the consumer easier. If you want help let me know.
Secor https://github.com/pinterest/secor has a nice configuration with wildcarding

[reachkrishnaraj]

@estezz : hmm, but we need to think about this scenario where,say only 1 consumer instance only has problem. In that case, we will need to bring all our other instances/threads down causing lag to in the indexing. This will be costly for index heavy consumers. I am think it will be good to implement both and provide an option in the config for folks to choose between option 1 and 2.

[estezz]

Both options would be awesome.