InsertOpts option to reconcile metadata where multiple are encountered

[brandur]

Here, attempt to resolve #165 by better defining how metadata that may
be inserted via InsertOpts on Insert will interact with metadata
that may be inserted via a job implementing InsertOpts(). Currently, a
job's metadata is always ignored.

The behavior to reconcile two metadata objects has some nuance, and I
don't think any one solution will fit everyone's desires. Here, I
propose that we let users defined their own preferred behavior by
implementing a new InsertOptsMetadataReconcile option:

type InsertOpts struct {
    ...

    // MetadataReconcile defines how, in the presence of multiple InsertOpts
    // like one from the job itself and another one from a Client.Insert param,
    // how metadata will be reconciled between the two. For example, two
    // metadata can be merged together, or one can exclude the other.
    //
    // Defaults to MetadataReconcileExclude.
    MetadataReconcile MetadataReconcile

The possible reconciliation modes:

• MetadataReconcileExclude: The more specific metadata (from Insert
  params) completely excludes the less specific metadata (from a job'
  opts), and the latter is discarded.

• MetadataReconcileMerge: Carries out a shallow merge. In case of
  conflict, keys from the more specific metadata win out.

• MetadataReconcileMergeDeep: Carries out a deep merge, recursing into
  every object to see if we can use it as a map[string]any. In case of
  conflict, keys from the more specific metadata win out.

The default setting is MetadataReconcileExclude. This is partly a
safety feature because it is possible to create oddly formed metadata
that might have trouble merging, and partly a performance feature.
Exclude doesn't require trying to parse any of the metadata so that we
can try to merge it.

A note on typing: I tried to marshal/unmarshal maps as map[any]any
instead of map[string]any so we could support all kinds of wild key
types simultaneously, but encoding/json is pretty strict about working
with only one type of key at a time, and failed when I tried. We may
have to keep it as an invariant that if you want to use one of the merge
modes, your metadata must be parseable as map[string]any. This
shouldn't be a problem because a struct with json tags on it will
always produce a compatible object. I've tried to document this.

Fixes #165.

[brandur]

@bgentry Wanted to get your thoughts on this one — I think the different reconcile options are kind of nice for maximizing flexibility, but there is a chance it's overkill. Thoughts?

[bgentry]

Hmm, I think it's partly a bit overkill because I'm not sure the other modes would be needed, but primarily I think the default is wrong and will be a footgun in terms of functionality we have planned. Am I reading correctly that by default if you set metadata in the worker InsertOpts that it will completely overwrite any other specified metadata?

[brandur]

So the reason I opted for the exclude option by default is that it certainly has some nice benefits:

• Merging is inherently much slower because it involves parsing two separate metadata hashes, gluing them together, then serializing them again. Especially where insertion speed is a concern, definitely nice not to have that going on in the background with every operation.
• Because we have to use map[string]any for the merge, it is possible for users to create incompatible metadata hashes that could produce an error (say using integers as keys for example).

I'm not sure I know the complete plans for future features well enough to see why merging is that important, so I may be missing something there.

I'd still argue strongly that the "exclude" mode needs to be in there as an option because a user should be able to clobber all secondary metadata hashes on an insert if they want to. I could see an argument for dropping the shallow merge option to lean things out, although it shouldn't be particularly harmful to keep in there either.