loadYamlContent(..) converts unset properties (should be: null) to empty strings ('')

[headcr4sh]

Bicep version
Bicep CLI version 0.27.1 (4b41cb6d4b)

Describe the bug
While using a combination of loadYamlContent() in conjunction with union() and string() to construct an OpenTelemetry Collector configuration file (don't ask...!) I stumbled upon a rather interesting bug:

yamlTest.bicep:

targetScope = 'subscription'

// expected: `{"toplevel":{"nested1":null,"nested2":null,"nested3":""}}`
// got: `{"toplevel":{"nested1":"","nested2":null,"nested3":""}}`
var yamlAsString = string(loadYamlContent('./example.yaml'))

example.yaml:

toplevel:
  nested1:
  nested2: null
  nested3: ''

To Reproduce
Steps to reproduce the behavior:
Create the two files as described above and use bicep to build the ARM template, e.g. by using the following shell script:

cat <<EOF > ./example.yaml
toplevel:
  nested1:
  nested2: null
  nested3: ''
EOF

cat <<EOF > ./yamlTest.bicep
targetScope = 'subscription'

// expected: '{"toplevel":{"nested1":null,"nested2":null,"nested3":""}}'
// got: '{"toplevel":{"nested1":"","nested2":null,"nested3":""}}'
#disable-next-line no-unused-vars
var yamlAsString = string(loadYamlContent('./example.yaml'))
EOF

az bicep build --file ./yamlTest.bicep --outfile ./yamlTest.out.json
jq -r '.variables' ./yamlTest.out.json

# Now: just compare the contents of the variable with the given YAML input

Additional context
Mitigation in my case was to explicitly set the properties to null, but it was kind of counter-intuitive and took me some time to figure out that Bicep's loadYamlContent() was doing something unexpected.

[ahelland]

From what I can tell after a quick debug lookup is that the Yaml deserialization is handled by a third-party library called SharpYaml (https://github.com/xoofx/SharpYaml/tree/master) which happens to treat an empty value the same as ''.

[headcr4sh]

I just had a look at the YAML 1.2.2 specification:

YAML allows the node content to be omitted in many cases. Nodes with empty content are interpreted as if they were plain scalars with an empty value. Such nodes are commonly resolved to a “null” value.

Example 7.3: Completely Empty Flow Nodes seems to further clarify how empty node content should be resolved.

I tried different YAML parsers (e.g. yq, pyyaml and Go's yaml/v2 package, which is based on LibYYAML) and it seems as if most parsers resolve empty node values to null, whereas SharpYaml seems indeed to be a notable exception which I assume handles things in an unexpected way which is not in line with the spec.

EDIT:
Raised an issue in the SharpYaml repo: #119

[anthony-c-martin]

Thanks for digging into this!