Add a facility for unlocked director iteration

[nigoroll]

Pondering #4140 made me realize that our central per-vcl director list constitutes a classic case of iterating a mutable list. That besides the already known fact that running potentially expensive iterator functions while holding the vcl_mtx is a very bad idea.

So this patch is a suggestion on how to get out of this situation. It does not go all the way (it still leaves unsolved a similar problem of iterating over all backends of all vcls), but shows an attempt on how to tackle the "iterate over one VCL's backends".

We add a fittingly named 'vdire' facility which manages the director list and, in particular, director retirement. The main change is that, while iterators are active on the director list, vcl_mtx is not held and any request of a director to retire only ends up in a resignation, which manifests by this director being put on a second list.

When all iterators have completed, the resignation list is worked and the actual retirement carried out.

NOTE: I am particularly not happy about the vdire_(start|end)_iter() interface being exposed. What I would like to see is a FOREACH macro which does both the start and end. Having to use an iterator with a callback is clumsy, and having to directly call start/end with a VTAILQ_FOREACH feels like crossing interfaces. See #4142 (comment)

[nigoroll]

After rebasing on the newly understood 4e2c50d, I have successfully run my stress test on this patch without any issues.

[nigoroll]

What I would like to see is a FOREACH macro which does both the start and end

Here's a macro which does that:

/*
 * this macro might look a bit daunting, but it is really just two things:
 *
 * - the outer for loop is only to declare the local "next" variable
 *   and execute the inner for loop once
 *
 * - the inner for loop is VTAILQ_FOREACH_SAFE with calls to
 *   vdire_{start,end}_iter() added at the right places:
 *   start is called at initialization and end when var becomes NULL
 *   and the loop ends
 *
 * problem: none of break, goto or return must be used from the loop body
*/

#define _VDIRE_FOREACH(var, vdire, next)				\
	for (struct vcldir *next = (void *)&next;			\
	     next == (void *)&next;					\
	     next = NULL)						\
		for (vdire_start_iter(vdire),				\
		     (var) = VTAILQ_FIRST(&(vdire)->directors);		\
		     ((var) || (vdire_end_iter(vdire), 0)) &&		\
		       ((next) = VTAILQ_NEXT(var, directors_list), 1);	\
		     (var) = (next))

#define VDIRE_FOREACH(var, vdire)				\
	_VDIRE_FOREACH(var, vdire, VUNIQ_NAME(_vdirenext))

As stated in the comment, none of break, goto or return must be used from within the loop body. So I think I prefer the explicit vdire_{start,end}_iter()

[nigoroll]

Putting this out of draft status despite not having worked on it any more, because it has received no feedback and I think it's actually important - now having seen one production case where this is at least a potential candidate.

[nigoroll]

bugwash: good idea by phk: New iterators encountering retired backends should wait until they are cleared, otherwise a constant stream of iterators could prevent retirement. I do not think this is a realistic scenario at this point, but we might as well do this right from the start.

another feedback question is if we can simplify this patch with an r/w lock. I am sure that I did look at this for quite some time and came to the conclusion that it does not do much to help us, but I will revisit this question.

[nigoroll]

New iterators encountering retired backends should wait until they are cleared

done: 41c42a0

another feedback question is if we can simplify this patch with an r/w lock.

I do not see the simplification. With an rwlock, vdire_resign() would try to wrlock and, if this fails, would need to start a thread which would then wrlock and do the remove.