MB-60971: Improve drain rate of the in-memory segments

[Thejas-bhat]

• Currently when there is a certain amount of data in memory, persister performs an in-memory merge and introduces the merged file as part of the new snapshot.
• This can have effects where the index is not able to ingest data because of an expensive ongoing in-memory merge operation which is not really bounded.
• The PR introduces the concept of persister workers and the max amount of data each worker can parallely perform an in-memory merge. The merged files are introduced to the system in a single step however to avoid having too many files in the system (thereby leading to pressure on the cleanup procedure which deletes these old zero-ref'd files)
• The original behaviour of merging all segments and flushing in one shot is still optionally supported though.