It has nothing to do with the hash lookup implementation, this is the
outcome of a cache lookup.

This is a prerequisite before moving waiting list ownership from objhead
to objcore.

Adding expired variants of miss and hitmiss outcomes will enable better
code organization, mainly the extraction of the objhead lookup step into
its own function.

This creates a separate objhead lookup after the hash lookup, and a
central location to process the objhead lookup result. This makes both
the lookup code and the lookup result processing code more compact and
easier to follow individually. In particular, the objhead lookup now
operates entirely under the objhead lock and has a much better signal
to noise ratio.

The advantages of the objhead owning the waiting list are that we need
less waiting list heads as they are shared across multiple objcores, we
get more frequent opportunities to rush waiting requests, and we avoid
one expensive part of the lookup, which is to find an objhead from a
request hash.

The downsides are the requirement to always perform a new lookup, which
increases contention on the objhead lock, and prevents expired but valid
objects to be considered fresh. This is another way to describe the
cache-control no-cache directive: a response considered fresh at fetch
time, but immediately stale afterwards.

Having the waiting list on the objcore means that we reenter the cache
lookup with the object candidate from the previous lookup, so we may
short-circuit the more expensive objhead lookup when the objcore is
compatible with the request rushing off the waiting list. The expensive
request hash lookup is still avoided, just like before, the objcore has
a reference to the objhead.

This creates a new first step in the lookup process:

- rush match (when coming from the waiting list)
- hash lookup (unless coming from the waiting list)
- objhead lookup (unless there was a rush hit)

This moves the infamous waiting list serialization phenomenon to the
vary header match. Knowing that a rushed object is guaranteed to lead to
a cache hit allows the rush policy to be applied wholesale, instead of
exponentially. Only requests incompatible with the objcore vary header
may reenter the waiting list, a scenario no different from the spurious
rush wakeups when operating on the objhead.

The waiting list operations are still guarded by the objhead lock. As it
stands currently, the exponential rush is indirectly systematic, since
the unusable objcore is rushed before the objhead lookup.

This is the first step towards implementing no-cache and private support
at the header field granularity.

Now that objcores own their waiting lists, they can no longer have
anything waiting for them at creation time.

This has the same effect, with slightly less ceremony.

The objcore is officially referenced by a request when it leaves its
waiting list during a rush. This allows migrating a waiting list from
one objcore to the next without the need to touch reference counters.

The hash_oc field will still have to be updated. This field must be
be kept to navigate from a request to its waiting list to implement
walking away from a waiting list.

Refs varnishcache#3835

This is very useful to manually verify the rush_exponent behavior.

After the migration of the waiting list to the objcore, the result was
that each request would rush more requests upon reembarking by dropping
the waited objcore reference before the objhead lookup.

The reference is now dropped after the objhead lookup, allowing for the
lookup outcome to be checked, either triggerring a complete rush when
the object is serviceable (modulus vary match) or moving the waiting
list to the next busy objcore.

This avoids the spurious wakeups caused by objhead rushes when waiting
lists were owned by objheads. This is however a missed opportunity when
there are multiple concurrent busy objcores. This could be alleviated
by linking busy objcores together but at this point this is already a
net improvement. The only way to get multiple concurrent busy objcores
on the same objhead would be through the hash_ignore_busy flag anyway.

Now that waiting lists are owned by objcores, the rush policy no longer
needs to leak everywhere. Either all the waiters are rushed at once for
cacheable objects, or according to rush_exponent.