Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

path expression to access array elements directly #30

Open
koebbingd opened this issue Aug 2, 2012 · 24 comments
Open

path expression to access array elements directly #30

koebbingd opened this issue Aug 2, 2012 · 24 comments

Comments

@koebbingd
Copy link

Is there a way to access an array element directly?
example:

arr [
  {ip= 192.168.7.1, name="Server 1"},
  {ip= 192.168.7.2, name="Server 2"},
  {ip= 192.168.7.3, name="Server 3"}
]

As far as I understand the library I first have to access the array by calling config.getObjectList("arr") and then iterating over the list to access the fields of my objects.
I'd like to have the possibility of calling config.getString("arr[0].ip") or something like that...

Thanks.

@viktorklang
Copy link
Contributor

What's the cost/benefit ratio of this feature proposal?

@havocp
Copy link
Collaborator

havocp commented Aug 2, 2012

not currently. you could do getConfigList("arr").get(0).getString("ip") or something though. hardcoding an index in a path string seems like it wouldn't be useful most of the time, so I'm not sure it'd be worth it... the existing way to do things doesn't seem that bad. saving a few characters once in a while vs adding a fair amount more code to support the more complex path expression syntax. don't know.

@viktorklang
Copy link
Contributor

Also, if someone wants to do it they could always create a bridge over to use XPATH anyway.
👎 I think it is something that should live outside of the Config lib as it has nothing to do with configuration.

@koebbingd
Copy link
Author

It would be just a convenience function...
...but I see one advantage:

If you have a complex array with many entries as a kind of default and want to override just one value, you could use this syntax too:

arr [
  {a: 1, b: 1, v:1}
  {a: 2, b: 2, v:2}
  {a: 3, b: 3, v:3}
  {a: 4, b: 4, v:4}
....
  {a: 30, b: 30, v:30}
]

"arr[5].v" = 7

Perhaps this use case makes the proposal more interesting?!

@viktorklang
Copy link
Contributor

There's really nothing preventing you from developing such an extension.

@havocp
Copy link
Collaborator

havocp commented Sep 20, 2013

Someone else was asking about this lately; but in the context of using it from inside the config file, not in code.
I don't know how much more code it would be in the lib, or how often people would use it, though.

@havocp
Copy link
Collaborator

havocp commented Dec 27, 2014

If we fix this then #160 could be made to work instead of throwing an exception, if we want.

@ratulmukh
Copy link

I think this feature would be really helpful.

I have the same problem that @havocp mentioned (in the context of using it from inside the config file, not in code) . In my case, there exists a base conf file that has an array. I would like my clients to provide their own conf file that would override certain elements of the array in the base conf file. Right now, I have no elegant way to fix it, and that is just too bad. :( Any suggestions?

@havocp
Copy link
Collaborator

havocp commented Aug 27, 2016

The best you can do right now is probably to build the array up from ${?foo} references and then people can define (or set to null) each value foo in order to have it in the array or not. They can also prepend or append elements of course.

@tr00per
Copy link

tr00per commented Aug 30, 2016

Any update on this? It's been 4 years.

@havocp
Copy link
Collaborator

havocp commented Aug 30, 2016

Nobody is employed to go through feature requests here. The ones that have been done are the ones that somebody who wanted the feature chose to work on, or chose to hire someone to work on.

If nobody ever wants this enough to work on it or pay for it, then it will never be done. It doesn't matter how many years people have thought it would be nice to have, if none of them thought it was important enough to work on.

Not trying to be grumpy, trying to explain how it works. Most open source projects work the same way.

@ratulmukh
Copy link

@havocp, Sorry to say but I mulled over your suggestion to use ${?foo}, but it seemed more like a band-aid and not very elegant, especially given the fact that this can be achieved very nicely if we use JSONPath with JSON files. I was using JSONPath before switching to Typesafe Config, and I had switched for a very strong reason (includes and comments).

My final solution right now is to use Typesafe Config to read and parse the conf files, convert them to strings, feed them into JSONPath, and do the necessary array access operations. There are some disadvantages and inconveniences, but this worked better for me overall.

Your other suggestion of implementing this feature and contributing a pull request is something I might do in the future. Sorry, but I just do not have enough time for it right now.

@movermeyer
Copy link

I've run into this same issue when I wanted to check for the existence of a key in a config:

My config has a list of objects with a given attribute. I'm then diff'ing two of these configs and trying to check if a given key exists in a third config.

Example pseudocode in the style of python/pyhocon:

from pyhocon import ConfigFactory
config1 = ConfigFactory.parse_file("file1.conf")
config2 = ConfigFactory.parse_file("file2.conf")
config3 = ConfigFactory.parse_file("file3.conf")

differences = diff(config1, config2)  # diff implemented elsewhere. Returns a list of keys that differ between the two configs. Signature: (ConfigTree, ConfigTree) -> [str]

for key in differences:
    if config3.get(key, None) is None:  # Defaults to None if the key isn't in config3
        raise Exception("Value of {0} differs between config1 and config2, but '{0}' doesn't exist in config3!".format(key))

While diff() can detect the difference in lists within the configs, there is no value for key that diff() can return that will work in this existence check.

I understand that no one is working on this right now, but if someone does, this use case might be interesting.

Currently, I have to extend the key syntax (I did my_list.[0].my_subkey, where 0 is the index within the list), return the extended key syntax from diff() and have special handling for such keys within my calling code. It would be nice to have a formally approved key syntax for referring to the indices of lists.

@ghost
Copy link

ghost commented Jun 6, 2018

Ho do I use
config.withValue(path, value)
without a path string to an array element?

I tried the most natural way you'd think:

my.config.array[0].value

@havocp
Copy link
Collaborator

havocp commented Jun 6, 2018

You would need to get the array, manually modify it, and then replace the entire array value

@ghost
Copy link

ghost commented Jun 6, 2018

I've been trying this, with config.processes being an array of objects, I did no changes to ml, yet it doesn't work

        val ml = config.getConfigList("config.processes") 
//... change ml
        val c = config.getConfig("config").withValue("processes", ConfigValueFactory.fromIterable(ml))

but it throws:

com.typesafe.config.ConfigException$BugOrBroken: bug in method caller: not valid to create ConfigValue from: Config(SimpleConfigObject({"days":1,"id":"CalendarKlubschule","offsetSec":5,"runEverySec":120,"serviceUrl":"http://localhost:9000/test/klubschule/%s%s%s"}))

at com.typesafe.config.impl.ConfigImpl.fromAnyRef(ConfigImpl.java:281)
at com.typesafe.config.impl.ConfigImpl.fromAnyRef(ConfigImpl.java:273)
at com.typesafe.config.impl.ConfigImpl.fromAnyRef(ConfigImpl.java:194)
at com.typesafe.config.ConfigValueFactory.fromAnyRef(ConfigValueFactory.java:72)
at com.typesafe.config.ConfigValueFactory.fromIterable(ConfigValueFactory.java:114)
at com.typesafe.config.ConfigValueFactory.fromIterable(ConfigValueFactory.java:151)
at Tests.testWebSockets(Tests.kt:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

@havocp
Copy link
Collaborator

havocp commented Jun 6, 2018

fromIterable is expecting plain Java objects but it looks like you have ConfigValue objects (or a mix, maybe a plain Java list with ConfigValue in it?) hard to say exactly without the ml-modification code.

@ghost
Copy link

ghost commented Jun 7, 2018

Ok managed to put it together. This replaces a global config I access, code is in Kotlin.
I use it when, during tests, I want to replace some config vars to make my app testable.

    fun replaceArrayValue(pathToArray:String, key:String, newValue:Any) {
        val cfgList = config.getConfigList(pathToArray)
        val replaceArray = cfgList.map {it.entrySet().associate {
            if(key == it.key)
                Pair(it.key, ConfigValueFactory.fromAnyRef(newValue))
            else
                Pair(it.key, it.value)
        }}

        config = config.withValue(pathToArray, ConfigValueFactory.fromIterable(replaceArray)).withFallback(config)
    }

and this is how you use it:

AdapterConfig.replaceArrayValue("config.processes", "serviceUrl", "newValue")

@vladp
Copy link

vladp commented Aug 18, 2020

I am also looking for a way to reference an array element within config.
Conceptually, I am trying to simplify configuration and allow users to use 'foreign keys' so to speak into another section.
Here is an example:


mainConfig= {

  defaults={
     allowedZones=["zone1,"zone2","zone3"]
  }

     zoneIntroScreen = ${mainConfig.defaults} {
       screensA=[
                { appliesToZone="zone1"
                  attr1="some parm1"
                  attr2="some attr2"},

                { appliesToZone="zone2"
                  attr1="some parm1"
                  attr2="some attr2"},
            ]
     } //end of zone intro
} //end of main config


In the appliesToZone attribute, I would like to reference a particular element of the allowedZones array
So that the user, would not make a mistake of typing in "ZONE 1" into appliedToZones values.. as an example.

@WayneWang12
Copy link

A simple use case is like:

Say we may have an environment variable or not(most likely it was an ipv6 address). When using IPV6 address, the config of akka hostname shold be [${?IPV6_ADDR}]. And if there is no IPV6, then we should use ${?IPV4_ADDR}.

To aquire a conditional config, I may use this:

real_addresses = [${?IPV6}, ${?IPV4}, "["${?IPV6}"]"]

akka {
  remote {
    artery {
      canonical {
        hostname = "0.0.0.0"      # external (logical) hostname
        hostname = ${?real_addresses.1}      # external (logical) hostname
 }
}

If ${?real_addresses.1} can be resolved, then we can switch smoothly between ipv4 and ipv6 and acheive a way to do conditional config.

So I think it is worthy of implementing such a function. @viktorklang

@viktorklang
Copy link
Contributor

@WayneWang12 Wouldn't you be able to do:

akka {
  remote {
    artery {
      canonical {
        hostname = "0.0.0.0"      # external (logical) hostname
        hostname = ${?IPV4}
        hostname = ${?IPV6}
 }
}

@WayneWang12
Copy link

WayneWang12 commented Nov 22, 2020

@WayneWang12 Wouldn't you be able to do:

akka {
  remote {
    artery {
      canonical {
        hostname = "0.0.0.0"      # external (logical) hostname
        hostname = ${?IPV4}
        hostname = ${?IPV6}
 }
}

No, because it fails to build up a connnection. IPV6 need to be a [fdbd:dc03:ff:1:1:27:9:14] to be parsed in akka. I may make a pull request to akka to solve this problem.

Play's ws client also need it. Details are here: RFC 2732

Url must be like:

  http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/index.html
  http://[1080:0:0:0:8:800:200C:417A]/index.html
  http://[3ffe:2a00:100:7031::1]
  http://[1080::8:800:200C:417A]/foo
  http://[::192.9.5.5]/ipng
  http://[::FFFF:129.144.52.38]:80/index.html
  http://[2010:836B:4179::836B:4179]

@viktorklang

@voodoo144
Copy link

I face the same issue today.

From my point of view HOCON provide this mechanism as "path" for referencing. It should support arrays as well.
Notation like ${array[0].array_member_prop} or ${array.0.array_member_prop}

@sd65
Copy link

sd65 commented Jun 24, 2021

We are facing the same issue today. Our config look like this :

key {
  nested-list = [
    { 
      nested-key = "demo"
    }
  ]
}

We tried to overwrite nested-key like this :

key.nested-list.0.nested-key
key.nested-list[0].nested-key

But as said before it does not work in hocon.

A similar mechanism do exists for java properties and env vars.

KilowattJunkie pushed a commit to KilowattJunkie/config that referenced this issue Feb 17, 2023
- Fixes lightbend#30
- Adds and modifies peek and find methods to allow indexing of ConfigList objects
- Enables nested array and object access using indices in paths:
        listConfig.getString("ofObject.0.byteObj.byteVal")
        listConfig.getString("ofArray.2.2")
KilowattJunkie pushed a commit to KilowattJunkie/config that referenced this issue Feb 18, 2023
- Fixes lightbend#30
- Adds and modifies peek and find methods to allow indexing of ConfigList objects
- Enables nested array and object access using indices in paths:
        listConfig.getString("ofObject.0.byteObj.byteVal")
        listConfig.getString("ofArray.2.2")
KilowattJunkie pushed a commit to KilowattJunkie/config that referenced this issue Feb 18, 2023
- Fixes lightbend#30
- Adds and modifies peek and find methods to allow indexing of ConfigList objects
- Enables nested array and object access using indices in paths:
        listConfig.getString("ofObject.0.byteObj.byteVal")
        listConfig.getString("ofArray.2.2")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants