inconsistency with escaping the commas

[julochrobak]

Bandicoot excepts and returns data in a CSV format. The CSV implementation in bandicoot is a very simple one and does not implement the RFC 4180. The double quote is not treated as a special character and only Unix end of line is supported. To allow commas or end of lines to be part of a string value, bandicoot uses the backslash as an escape character:

col1 string,col2 string
Hello World\,\
This is bandicoot,another string column value

col1 value is:

Hello World,
This is bandicoot

col2 value is:

another string column value

There is an inconsistency in terms of using the escape character between the HTTP req/resp body, HTTP URL (query), and the language literals.

The HTTP resp/req body always requires a comma to be escaped. If a comma is not escaped within an HTTP req body, the HTTP call fails with HTTP 400 complaining about invalid number of attributes in a tuple. However, there is no check on the HTTP URL (query) nor on the language literals. This means the following is a valid function call:

curl 'http://localhost:12345/Test?s=Hello%20World,'

Bandicoot won't complain but it also won't escape the comma. There are two consequences:

1.Inconsistency between the values of an attribute and primitive type values

fn Test(s string, in {ss string}) {ss string}
{
    return select (s == ss) (in);
}

this example has two different behaviors:

# the 's' variable contains non-escaped comma 's=Hello%20World,'
$ echo -e 'ss string\nHello World\,' | \
    curl --data-binary @- 'http://localhost:12345/Test?s=Hello%20World,'
ss string

# the 's' variable contains properly escaped comma 's=Hello%20World\,'
$ echo -e 'ss string\nHello World\,' | \
    curl --data-binary @- 'http://localhost:12345/Test?s=Hello%20World\,'
ss string
Hello World\,

The same problem happens with using a litter in the function:

fn Test(in {ss string}) {ss string}
{
    return select ("Hello World," == ss) (in);
}

or

fn Test(in {ss string}) {ss string}
{
    return select ("Hello World\," == ss) (in);
}

2.There is also one more serious problem related to allowing non-escaped commas. If there is a tuple with a string value which has a non-escaped comma, the CSV result is invalid

fn Test(in {orig string}) {orig, new string}
{
    return extend (new = "Here is a comma without a backslash: ,") (in);
}

$ echo -e 'orig string\nHello World\,' |curl --data-binary @- 'http://localhost:12345/Test'
new string,orig string
Here is a comma without a backslash: ,,Hello World\,