Template variables are strings included in the JSON of a template that access properties of task components. They take the following form where VARIABLE is the variable being used:
{% VARIABLE %}
Expansion is the process by which a variable is converted from the form above to the actual value it represents. Once expansion completes a variable is referred to as expanded. The following template sections support the expansion of template variables:
archivescontextstestsreferenceobject oftasks
Note
Sections such as addresses, groups, hosts and fields outside the reference section in tasks DO NOT support template variables and values will not be expanded in those sections if used.
When a template variable is used, it must be enclosed within quotes so that a pre-expansion template is still valid JSON. Once expanded, whether the expanded variable is still a JSON string or some other JSON type, depends on two factors:
- The variable used
- Whether the variable is used standalone or embedded within a string
A standalone use of the variable is when you use just the variable surrounded by quotes. For example:
{
"example": "{% flip %}"
}
In this example, since the :ref:`flip variable <psconfig_templates_vars-flip>` is a boolean type and we have used it in standalone mode, the object will be expanded to the following (assuming flip evaluates to true):
{
"example": true
}
Notice that the surrounding double quotes are removed so it is a JSON boolean instead of a string. If we were to use a different variable, like :ref:`scheduled_by_address <psconfig_templates_vars-scheduled_by_address>` that always evaluates to a string, the double quotes will remain intact. For example, consider the following template:
{
"example": "{% scheduled_by_address %}"
}
If we expand that variable we get the following (assuming 10.0.0.1 is the address to which it evaluates):
{
"example": "10.0.0.1"
}
Notice that since the variable is a string, the quotes are preserved.
pSConfig agents can also expand variables embedded within an existing string. For example if we want to build a URL with a dynamic host portion that uses :ref:`scheduled_by_address <psconfig_templates_vars-scheduled_by_address>` we can do the following:
{
"example": "https://{% scheduled_by_address %}/example/url"
}
This would expand to:
{
"example": "https://10.0.0.1/example/url"
}
For those objects that support template variables, they can be nested as deeply within the object as needed. For example, all the uses of template variables below are completely valid:
{
"example": "{% scheduled_by_address %}",
"nest1": {
"example": "{% scheduled_by_address %} is nested one level",
"nest2": {
example": "{% scheduled_by_address %} is nested two levels"
}
}
}
Assuming :ref:`scheduled_by_address <psconfig_templates_vars-scheduled_by_address>` still evaluates to 10.0.0.1, the expanded object would look as follows:
{
"example": "10.0.0.1",
"nest1": {
"example": "10.0.0.1 is nested one level",
"nest2": {
example": "10.0.0.1 is nested two levels"
}
}
}
This variable accesses the address property of an address object in the list generated by a group for an individual task.
Note
See :doc:`psconfig_templates_intro` for a discussion on how groups work and how they generate address lists.
The address template variable takes the following form:
{% address[INDEX] %}
The INDEX between square brackets indicates which address object in the generated list to access starting at 0. Since groups of type mesh and disjoint always return pairs of two addresses, the INDEX can be value 0 or 1. For groups of type list it must be 0 since that type only generates one address. This variable always returns a string and will take the form of an IP address or hostname.
The address template variable is commonly used for test specifications that have source and/or dest parameters. Below is an example of a throughput test using this variable to set the source to the first address in the pair and the dest to the second address. Assuming that the first address in the input pair is 10.0.0.1 and the second is 10.0.0.2 the template and expansion look as follows:
Template:
"throughput_test": { "type": "throughput", "spec": { "source": "{% address[0] %}", "dest": "{% address[1] %}", } }Post-Expansion:
"throughput_test": { "type": "throughput", "spec": { "source": "10.0.0.1", "dest": "10.0.0.2", } }
Likewise, this variable is useful for archive objects such as those of type http where the URL is to an archive running on one of the endpoints. For example, if paired with the test specification above, this would always store the result on the host that is the source of the test. The template and expanded values are shown below:
Template:
"source_archive": { "archiver": "http", "data": { "_url": "https://{% address[0] %}/logstash" } }Post-Expansion:
"source_archive": { "archiver": "http", "data": { "_url": "https://10.0.0.1/logstash" } }
The flip variable expands a boolean value that is true if the task will be scheduled by an agent other than the one representing either the first address in the list or the host indicated by the scheduled_by parameter of the task object if specified. What this usually means is that if the first address in a pair has :ref:`no-agent <psconfig_templates_advanced-addresses-noagent>` enabled then the second address will have to schedule it, thus setting flip to true. Otherwise flip will be false.
The flip template variable takes the following form:
{% flip %}
The return type of this is a JSON boolean meaning if used as a standalone variable the surrounding double quotes will be removed.
The flip variable is commonly used with the aptly-named flip option of latency and latencybg test specifications. In fact, it is probably a good idea to set this for all tests of this type. What it does is allow these tests to work even if the the address set to the source has :ref:`no-agent <psconfig_templates_advanced-addresses-noagent>` enabled. Assuming that the first address in the input pair is 10.0.0.1 , the second is 10.0.0.2 and the address object of 10.0.0.1 has :ref:`no-agent <psconfig_templates_advanced-addresses-noagent>` enabled, the template and expansion look as follows:
Template:
"latencybg_test": { "type": "latencybg", "spec": { "source": "{% address[0] %}", "dest": "{% address[1] %}", "flip": "{% flip %}" } }Post-Expansion:
"latencybg_test": { "type": "latencybg", "spec": { "source": "10.0.0.1", "dest": "10.0.0.2", "flip": true } }
This variable accepts a jq script that it performs against the JSON of a task and its components. This is by far the most flexible and powerful of the template variables. In fact, almost all the other template variables could be implemented using the jq template variable instead.
The jq template variable takes the following form:
{% jq SCRIPT %}
The SCRIPT is the jq to be executed. The JSON object on which the jq operates is a special form of the template itself, including only those elements relating to the individual task being generated. Specifically, the JSON being queried has the following sections:
addresseswhich is an array of the address object for this particular task. This means a variable like{% jq .addresses[0].address %}is the exact equivalent of the{% address[0] %}template variable (see :ref:`psconfig_templates_vars-address`).archivesis an array of the archive objects to be used for this task. A query such as{% jq .archives[0].archiver %}returns the type of the first archiver in the list. The order is not guaranteed, so plan scripts accordingly if using more than one archiver for a task.contextsis a two-dimensional array of the context objects to be used for this task. The index at the first level of the array corresponds with the index of the address object that is associated with the context objects in the list at the second level. A query such as{% jq .contexts[0] %}returns an array of all the context objects associated with the first address object. The order within the second level of the context list is not guaranteed, so plan scripts accordingly if using more than one context for an address. If an address has no contexts, then the array will be empty.hostsis an array of the host objects to be used for this task. The index of a host maps to the address to which that host belongs. If an address does not belong to a host, an empty object will be at that position. A script such as{% jq .host[0].tags %}returns the tags of the host associated with the first address.taskis the task object exactly as defined in the template. A script such as{% jq .task.tools %}returns thetoolsarray of the task.testis the test object exactly as defined in the template. A script such as{% jq .test.type %}returns thetypeof the task.
The JSON type returned by jq template variables depends on your query. It is important to keep this in mind when using these variables because the cost of the extra flexibility means unexpected things can happen if one is not careful.
Let's take a look at an example where we have the following template using the jq template variable in the reference section of a task object:
{
"addresses": {
"thr1": {
"address": "thr1.perfsonar.net",
"_meta": {
"ifspeed": 10
}
},
"thr2": {
"address": "thr2.perfsonar.net",
"_meta": {
"ifspeed": 1
}
}
},
"groups": {
"throughput_group": {
"type": "mesh",
"addresses": [
{"name": "thr1"},
{"name": "thr2"}
]
}
},
"tests": {
"throughput_test": {
"type": "throughput",
"spec": {
"source": "{% address[0] %}",
"dest": "{% address[1] %}"
}
}
},
"tasks": {
"throughput_task": {
"group": "throughput_group",
"test": "throughput_test",
"reference": {
"source_ifspeed": "{% jq .addresses[0]._meta.ifspeed %}",
"dest_ifspeed": "{% jq .addresses[1]._meta.ifspeed %}"
}
}
}
}
In the example, each address object has a custom _meta property intended to indicate the speed of the network interface that the address object represents. Those fields will be included in a reference section of the task meaning it will be included in an informational section of the same name in the corresponding pScheduler task. Someone debugging pScheduler will be able to see these fields and may find it of use. Let's take a closer look at how these will be expanded so we better understand their meaning.
First, we have a group of type mesh containing two members. This will generate the following address pairs:
- thr1.perfsonar.net, thr2.perfsonar.net
- thr2.perfsonar.net, thr1.perfsonar.net
For the first address pair, the JSON generated against which we can run our jq script is shown below:
{
"addresses":[
{
"address":"thr1.perfsonar.net",
"_meta":{
"ifspeed":10
}
},
{
"address":"thr2.perfsonar.net",
"_meta":{
"ifspeed":1
}
}
],
"archives": [],
"contexts": [ [], [] ],
"hosts": [ {}, {} ],
"test":{
"type":"throughput",
"spec":{
"source":"{% address[0] %}",
"dest":"{% address[1] %}"
}
},
"task":{
"group":"throughput_group",
"test":"throughput_test",
"reference":{
"source_ifspeed":"{% jq .addresses[0]._meta.ifspeed %}",
"dest_ifspeed":"{% jq .addresses[1]._meta.ifspeed %}"
}
}
}
Our variables {% jq .addresses[0]._meta.ifspeed %} and {% jq .addresses[1]._meta.ifspeed %} are standalone queries selecting a JSON integer from each of the address object in the list. That means the quotes will get dropped in the result. This yields the following expanded task:
{
"group":"throughput_group",
"test":"throughput_test",
"reference":{
"source_ifspeed":10,
"dest_ifspeed":1
}
}
For the second address pair, we get the following JSON against which we can run our jq script:
{
"addresses":[
{
"address":"thr2.perfsonar.net",
"_meta":{
"ifspeed":1
}
},
{
"address":"thr1.perfsonar.net",
"_meta":{
"ifspeed":10
}
}
],
"archives": [],
"contexts": [ [], [] ],
"hosts": [ {}, {} ],
"test":{
"type":"throughput",
"spec":{
"source":"{% address[0] %}",
"dest":"{% address[1] %}"
}
},
"task":{
"group":"throughput_group",
"test":"throughput_test",
"reference":{
"source_ifspeed":"{% jq .addresses[0]._meta.ifspeed %}",
"dest_ifspeed":"{% jq ,addresses[1]._meta.ifspeed %}"
}
}
}
This results in the following expanded task:
{
"group":"throughput_group",
"test":"throughput_test",
"reference":{
"source_ifspeed":1,
"dest_ifspeed":10
}
}
Note
For more information on jq, see the official jq documentation or this video tutorial.
This variable accesses the lead-bind-address property of an address object in the list generated by a group for an individual task. It will additionally fallback to the value of the address property if lead-bind-address is not set.
Note
See :doc:`psconfig_templates_intro` for a discussion on how groups work and how they generate address lists.
Note
See :ref:`psconfig_templates_advanced-addresses-lead_bind_address` for a discussion of the lead-bind-address property's meaning
The lead_bind_address template variable takes the following form:
{% lead_bind_address[INDEX] %}
The INDEX between square brackets indicates which address object in the generated list to access starting at 0. Since groups of type mesh and disjoint always return pairs of two addresses, the INDEX can be value 0 or 1. For groups of type list it must be 0 since that type only generates one address. This variable always returns a string and will take the form of an IP address or hostname.
This property is primarily available for completeness and informational purposes. The pScheduler agent reads the address object's lead-bind-address property automatically, so it is not required to use this variable in order to properly set a pScheduler server's binding options. The example below shows lead_bind_address being used for informational purposes in a task:
{
"group":"throughput_group",
"test":"throughput_test",
"reference":{
"source_lead_bind_address": "{% lead_bind_address[0] %}"
}
}
Assuming the lead-bind-address property is set to 10.0.0.1 OR that it is not set and the address property is 10.0.0.1 we get the following:
{
"group":"throughput_group",
"test":"throughput_test",
"reference":{
"source_lead_bind_address": "10.0.0.1"
}
}
The localhost template variable returns a string value of "localhost" unless the :ref:`flip <psconfig_templates_vars-flip>` variable is true, then it returns the value of :ref:`scheduled_by_address <psconfig_templates_vars-scheduled_by_address>`.
The localhost template variable takes the following form:
{% localhost %}
This variable may be useful in instances where :ref:`scheduled_by_address <psconfig_templates_vars-scheduled_by_address>` is desired, but the local system identifies itself using a private IP address different from that in :ref:`scheduled_by_address <psconfig_templates_vars-scheduled_by_address>`.
Note
If you do not have the private IP limitation, then :ref:`scheduled_by_address <psconfig_templates_vars-scheduled_by_address>` is generally preferred as its clearer as to where results are stored once expanded.
The localhost template variable can be useful when building URLs for archives. The example below embeds the variable in the url field. If the host expected to schedule the test does not have :ref:`no-agent <psconfig_templates_advanced-addresses-noagent>` enabled then it will expand to "localhost" as shown below:
Template:
"local_archive": { "archiver": "http", "data": { "_url": "https://{% localhost %}/logstash" } }Post-Expansion:
"local_archive": { "archiver": "http", "data": { "_url": "https://localhost/logstash" } }
If that same example is for a task where the host expected to schedule the task has :ref:`no-agent <psconfig_templates_advanced-addresses-noagent>` enabled, thus :ref:`flip <psconfig_templates_vars-flip>` is true, then it will return the following if the address of the :ref:`no-agent <psconfig_templates_advanced-addresses-noagent>` host is 10.0.0.1:
Post-Expansion:
"local_archive": { "archiver": "http", "data": { "_url": "https://10.0.0.1/logstash" } }
This variable accesses the pscheduler-address property of an address object in the list generated by a group for an individual task. It will additionally fallback to the value of the address property if pscheduler-address is not set.
Note
See :doc:`psconfig_templates_intro` for a discussion on how groups work and how they generate address lists.
Note
See :ref:`psconfig_templates_advanced-addresses-pscheduler_address` for a discussion of the pscheduler-address property's meaning
The pscheduler_address template variable takes the following form:
{% pscheduler_address[INDEX] %}
The INDEX between square brackets indicates which address object in the generated list to access starting at 0. Since groups of type mesh and disjoint always return pairs of two addresses, the INDEX can be value 0 or 1. For groups of type list it must be 0 since that type only generates one address. This variable always returns a string and will take the form of an IP address, hostname with an optional port.
Most of the standard perfSONAR pScheduler test plug-ins support a source-node and/or dest-node field in their specification. These are for defining a pScheduler server on a port other than 443 or at a different address from the source and dest fields. These are exactly the types of fields for which pscheduler_address is intended. Let's assume we have a task using the following pair of address definitions:
"addresses":[
{
"address":"thr1.perfsonar.net",
"pscheduler-address": "[fd89:b4d9:341a:8465::1]:8080"
},
{
"address":"thr2.perfsonar.net",
}
]
Below is test object below and its subsequent expansion using the address objects above:
Template:
{ "type":"throughput", "spec":{ "source":"{% address[0] %}", "dest":"{% address[1] %}", "source-node":"{% pscheduler_address[0] %}", "dest-node":"{% pscheduler_address[1] %}" } }Post-Expansion:
{ "type":"throughput", "spec":{ "source":"thr1.perfsonar.net", "dest":"thr2.perfsonar.net", "source-node":"[fd89:b4d9:341a:8465::1]:8080", "dest-node":"thr2.perfsonar.net" } }
Notice how {% pscheduler_address[0] %} expands to the pscheduler-address property of the first address, but for {% pscheduler_address[1] %} it falls back to the value of address since pscheduler-address is not set for the second address.
The scheduled_by_address variable expands to the address property of the address object responsible for creating this task. Responsibility is determined according to the following rules:
- If :ref:`scheduled-by <psconfig_templates_advanced-tasks-scheduled_by>` is set in the
task, then the generatedaddressobject at the index specified by that property is used unless it has :ref:`no-agent <psconfig_templates_advanced-addresses-noagent>` enabled - Otherwise, the first address in the generated list where :ref:`no-agent <psconfig_templates_advanced-addresses-noagent>` is disabled will be used.
Note
There will never be a task where all addresses have :ref:`no-agent <psconfig_templates_advanced-addresses-noagent>` enabled. pSConfig skips those tests since by definition they cannot be created.
The scheduled_by_address template variable takes the following form:
{% scheduled_by_address %}
This variable always returns a string and will take the form of an IP address or hostname.
This is often used in both the url and observer or measurment agent fields for archives (i.e. a field indicating what host performed the measurement). In the _url it tells it to register the results to the host requesting the task. In the x-ps-observer header it explicitly defines the host that requested the measurement, which is a special field the perfSONAR Logstash pipeline that writes to OpenSearch understands. Assuming the variable expands to 10.0.0.1, below is an example template and its expansion:
Template:
"sched_by_archive": { "archiver": "http", "data": { "_url": "https://{% scheduled_by_address %}/logstash" }, "_headers": { "x-ps-observer": "{% scheduled_by_address %}" } }Post-Expansion:
"sched_by_archive": { "archiver": "http", "data": { "_url": "https://10.0.0.1/logstash" }, "_headers": { "x-ps-observer": "10.0.0.1" } }