allow value-concatenating an object to each list element

[carrot-garden]

0. this is a feature request

1. use case:

• imaging you have network service with some 100 end points;
• each end point has some 30 parameters, mostly common to all points;
• may be 2 to 5 of the parameters are to be overridden by each point;

2. the way I understand current mechanisms of HOCON
   https://github.com/typesafehub/config/blob/master/HOCON.md
   this would make this definition very verbose;

3. how about new/proposed object$builder syntax:

//
// the list definition below will be looking for this builder
//
netty_point_list$builder {
   id = "you-must-override"
   localAddress = "1.2.3.4:12345"
   remoteAddress = "you-must-override"
   packetTTL = 5
   pipeline = "default-pipeline"
   pipeline_param_0 = 123
   // 20 more parameters here; mostly come from include  or variable substitution
}

//
// netty_point_list will be looking for optional netty_point_list$builder
// to initialize default list items parameters before override
//
netty_point_list [
   { id = "service-1", remoteAddress = "1.1.1.1:12345", pipeline = "advanced" }
   { id = "service-2", remoteAddress = "1.1.1.2:12345", localAddress = "3.3.3.3:12345" }
   // 100 more service definitions with little variations on the builder-defined parameters
]

4. am I missing some HOCON feature or what? :-)

[carrot-garden]

for those interested, a solution:

https://github.com/carrot-garden/carrot-conf/tree/master/carrot-conf-list

[viktorklang]

Won't withFallBack work here just fine?

[carrot-garden]

yes, but you must tell withFallBack where to get it from;

if you have a lot of lists, would it not make sense to have a convention with a list builder?

convention rules:

• if it is a list, look for a builder first
• look on the same level as list only
• look for builder name pattern "${list-name}/builder"
• if builder present, use it withFallBack(builder) on each list item;

[viktorklang]

I'd be surprised if it was more than 5 lines of code to create a method that does that for any ConfigMap.

[carrot-garden]

you are right; and this argument can be extended to include all of jdk :-)

but actually it is more then 5 lines
https://github.com/carrot-garden/carrot-conf/tree/master/carrot-conf-list

[havocp]

The way you can have a fallback for each list item right now is something like this:

mydefaults = {
 a=1, b=2
}

mylist = [
  { a = 3 } ${mydefaults}
  { b=10 } ${mydefaults}
]

In the spec this is "value concatenation"

it might be nice to avoid the repetition of ${mydefaults} after each list entry, but a syntax that feels elegant and general (maybe covers other similar cases? not sure) would need some thought.

You can also do this in code of course; when you load that list, map _.withFallback(getConfig("mydefaults")) over the list.

At some point this is supposed to be a (relatively) obvious declarative file, so I'm reluctant to start making it into a mini programming language...

I believe right now that concatenating/merging lists and objects is an error:

  [ a, b ] { foo: bar}  // error (I _think_), concat list with object

Viktor suggested the object in that case could be interpreted as "merge with each list element." One issue here is what happens if you have multiple lists in multiple places for the key, like say you have in reference.conf:

  [] { foo: bar}

Then in application.conf, someone wants to merge in the defaults to their list, how does that happen... you'd sort of need that {foo: bar} to be available as a substitution, so reference.conf would need to do:

mydefaults = { foo: bar}
mylist = [] ${mydefaults}

and application.conf:

mylist = [ a, b ] ${mydefaults}

but then that's getting a little tricky to explain to people.

If I were expecting third parties to edit application.conf, I'd probably go with the withFallback in code just so it'd be nice and easy to use.

[carrot-garden]

wow, you have a lot of cool ideas, thanks for sharing;

"value concatenation" is way too verbose; but this one is my favorite:

mylist = [ {a=1}, {b=2} ] ${mydefaults}

I hope you can accept it?

[viktorklang]

In the case of multiple defaults just apply them all?

[havocp]

If this is part of the existing "value concatenation" mechanism then multiples are already defined I guess. Value concatenation lets you have as many values as you want (a bunch of adjacent strings => one big concatenated string, a bunch of adjacent lists => one big concatenated list, a bunch of adjacent objects => one big merged object).
The idea here would extend the rule to make adjacent list+object do something instead of being an error.

One thing I just thought of, currently value concatenation is associative in the mathematical sense because concatenation and merging are both associative. So it doesn't matter if the concatenation "operator" (syntactic adjacency) is left or right associative. However in this list+object case, it gets a little confusing. []{}[]{} if left associative would fallback each element in the first list to the object in position 2, then concatenate the list in position 3, then fallback each element in the resulting list to the object in position 4. If right associative, it would fallback each element of the list in position 3 to the object in position 4, then I guess puke because an object on the left and list on the right wouldn't make sense.

Multiple objects just works with either associativity; []{}{} would either fallback each element to position 2 object, then fallback each element to position 3 object; OR merge the two objects, then fallback each element in the list to the merged object. Those should be equivalent so it doesn't matter.

This doesn't really give you a way to specify defaults in the sense that reference.conf specifies defaults. It just gives you a shorthand. Because the "concatenation" trick is a purely local transformation. If reference.conf happens to build its list with the []{} syntax, it still is just making a list. Later attempts to append to that list would not be affected (i.e. they wouldn't get the "defaults").

Another thought, this might naturally generalize such that a list of strings followed by a string value-concats the string to each element, but it breaks with a list of lists followed by a list since that already means something... inconsistent.

Another line of thought here would be to have some explicit "map" operator which would maybe be clearer, or maybe not, since I don't really know how that operator would be written (the word map? starts to be uncomfortably programming-language-like). But it avoids the inconsistency in the case of wanting to "map" a list onto a list of lists.

I don't know. I'm thinking we keep thinking about it and it's sort of a post-1.0 thing...

[carrot-garden]

wow! this reads like scala type system spec now :-)

can we get a taste of some simple interpretation of

mylist = [ {a=1}, {b=2} ] ${mydefaults}

make adjacent list+object do something instead of being an error

for v 0.6 ?

just to let more people see it and possibly share their thoughts?

[viktorklang]

Please feel free to take a stab at it!

[carrot-garden]

ok, here is my stab:

1. this already works, right:

mydefaults = {
 a=1, b=2
}
mylist = [
  { a=3  } ${mydefaults}
  { b=10 } ${mydefaults}
]

2. then do not throw any errors, just expand

mydefaults = {
 a=1, b=2
}
mylist = [
  { a=3  } 
  { b=10 } 
] ${mydefaults}

into

mydefaults = {
 a=1, b=2
}
mylist = [
  { a=3  } ${mydefaults}
  { b=10 } ${mydefaults}
]

and process as before

[viktorklang]

Do you have code + tests?

[carrot-garden]

yes

#31

[havocp]

for additional discussion see #31