Report

Version

The verions of the packages tested in this report.

	Version
toml	0.10.2
tomli/tomli_w	2.0.1; tomli_w: 1.0.0
tomlkit	0.12.5
rtoml	0.11.0
qtoml	0.3.1
tomllib	(Python 3.12.2)

Dumping `None` value

How the package dumps None value in python

Literally <package>.dumps(None)

	Dumped value or error
toml	'NoneType' object is not iterable
tomli/tomli_w	'NoneType' object has no attribute 'items'
tomlkit	Expecting Mapping or TOML Container, <class 'NoneType'> given
rtoml	"null" --- rtoml v0.11+ supports dumping None to a desired string: `rtoml.dumps(data, none_value='@None')`: "@None"
qtoml	'NoneType' object has no attribute 'items'
tomllib	module 'tomllib' has no attribute 'dumps'

Dumping key-`None` pair

How the package dumps key-value pair with value None

Literally <package>.dumps({"key": None})

	Dumped value or error
toml
tomli/tomli_w	Object of type <class 'NoneType'> is not TOML serializable
tomlkit	Invalid type <class 'NoneType'>
rtoml	key = "null" --- rtoml v0.11+ supports dumping None to a desired string: `rtoml.dumps(data, none_value='@None')`: key = "@None"
qtoml	TOML cannot encode None
tomllib	module 'tomllib' has no attribute 'dumps'

Dumping list with `None` value

How the package dumps a list with None value in it.

Literally <package>.dumps({"key": [1, 2, 3, None, 5]})

	Dumped value or error
toml	key = [ 1, 2, 3, "None", 5,]
tomli/tomli_w	Object of type <class 'NoneType'> is not TOML serializable
tomlkit	Invalid type <class 'NoneType'>
rtoml	key = [1, 2, 3, "null", 5] --- rtoml v0.11+ supports dumping None to a desired string: `rtoml.dumps(data, none_value='@None')`: key = [1, 2, 3, "@None", 5]
qtoml	bad type '<class 'NoneType'>' for dump_value
tomllib	module 'tomllib' has no attribute 'dumps'

Loading `None`-like values

How the package loads None-like value in string

Literally <package>.loads('v1 = "null" v2 = "None"')

	Loaded as
toml	{'v1': 'null', 'v2': 'None'}
tomli/tomli_w	module 'tomli_w' has no attribute 'loads'
tomlkit	{'v1': 'null', 'v2': 'None'}
rtoml	{'v1': 'null', 'v2': 'None'} --- rtoml v0.11+ supports loading custom None values: `rtoml.loads(data, none_value='None')`: {'v1': 'null', 'v2': None} `rtoml.loads(data, none_value='null')`: {'v1': None, 'v2': 'None'}
qtoml	{'v1': 'null', 'v2': 'None'}
tomllib	{'v1': 'null', 'v2': 'None'}

Dumping a heterogenous array

How the package dumps a python dictionary with a heterogenous array.

Literally <package>.dumps({"v": [1, 1.2, True, "string"]})

	Dumped value or error
toml	v = [ 1, 1.2, true, "string",]
tomli/tomli_w	v = [ 1, 1.2, true, "string", ]
tomlkit	v = [1, 1.2, true, "string"]
rtoml	v = [1, 1.2, true, "string"]
qtoml	v = [1, 1.2, true, 'string']
tomllib	Dumping not supported

Loading a heterogenous array

How the package loads a toml string with a heterogenous array.

Literally <package>.loads('v = [1, 1.2, True, "string"]')

	Loaded as
toml	Not a homogeneous array (line 2 column 1 char 1)
tomli/tomli_w	`{'v': [1, 1.2, True, 'string']}`
tomlkit	`{'v': [1, 1.2, True, 'string']}`
rtoml	`{'v': [1, 1.2, True, 'string']}`
qtoml	`{'v': [1, 1.2, True, 'string']}`
tomllib	`{'v': [1, 1.2, True, 'string']}`

Dumping a nested array

How the package dumps a python dictionary with a nested array.

Literally <package>.dumps({"v": [[1], [1, 2]]})

	Dumped value or error
toml	v = [ [ 1,], [ 1, 2,],]
tomli/tomli_w	v = [ [ 1, ], [ 1, 2, ], ]
tomlkit	v = [[1], [1, 2]]
rtoml	v = [[1], [1, 2]]
qtoml	v = [[1], [1, 2]]
tomllib	Dumping not supported

Loading a nested array

How the package loads a toml string with a nested array.

Literally <package>.loads('v = [[1], [1, 2]]')

	Loaded as
toml	`{'v': [[1], [1, 2]]}`
tomli/tomli_w	`{'v': [[1], [1, 2]]}`
tomlkit	`{'v': [[1], [1, 2]]}`
rtoml	`{'v': [[1], [1, 2]]}`
qtoml	`{'v': [[1], [1, 2]]}`
tomllib	`{'v': [[1], [1, 2]]}`

Dumping keeps order of keys?

Whether the package preserves the order of the keys while dumps a python dictionary.

Thus, whether <package>.dumps({"c": 1, "a": 2, "b": 3}) yields a string like c = 1\na = 2\nb = 3\n.

	Order kept?
toml	Kept
tomli/tomli_w	Kept
tomlkit	Kept
rtoml	Kept
qtoml	Kept
tomllib	Dumping not supported

Loading keeps order of keys?

Whether the package preserves the order of the keys while loads a TOML string.

Thus, whether <package>.loads('c = 1\na = 2\nb = 3\n') yields a dictionary with keys in the order of ['c', 'a', 'b'].

	Order kept?
toml	Kept
tomli/tomli_w	Kept
tomlkit	Kept
rtoml	Kept
qtoml	Kept
tomllib	Kept

Dumping unicode

How the package dumps Unicode in python

Literally, <package>.dumps({"你好": "世界"})

	Dumped value
toml	"你好" = "世界"
tomli/tomli_w	"你好" = "世界"
tomlkit	"你好" = "世界"
rtoml	"你好" = "世界"
qtoml	'你好' = '世界'
tomllib	Dumping not supported

Loaded unicode

How the package loads a file with unicode.

The file was created by:

# Create a file with unicode content
with open(self.datafile, "w", encoding="utf-8") as f:
    f.write('"你好" = "世界"\n')

# Use `<package>.load()` to load the file
with open(self.datafile, "r", encoding="utf-8") as f:
    loaded = <package>.load(f)

	Loaded as
toml	{'你好': '世界'}
tomli/tomli_w	File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')` When loaded with `rb`: {'你好': '世界'}
tomlkit	{'你好': '世界'}
rtoml	{'你好': '世界'}
qtoml	{'你好': '世界'}
tomllib	File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')` When loaded with `rb`: {'你好': '世界'}

Compliance with valid tests in toml-test

Test the compliance with the standard test suites for valid toml files here:

https://github.com/BurntSushi/toml-test/

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

	Result (toml-test v1.3.0)
toml	datetime/local-time.toml Parsed as unexpected data. datetime/datetime.toml Parsed as unexpected data. comment/tricky.toml Parsed as unexpected data. key/dotted.toml Found invalid character in key name: '"'. Try quoting the key name. (line 12 column 11 char 245) key/escapes.toml Parsed as unexpected data. string/escape-esc.toml Reserved escape sequence used (line 1 column 1 char 0) float/zero.toml Weirdness with leading zeroes or underscores in your number. (line 4 column 1 char 47) array/mixed-int-string.toml Not a homogeneous array (line 1 column 1 char 0) array/nested-double.toml Not a homogeneous array (line 1 column 1 char 0) array/mixed-int-float.toml Not a homogeneous array (line 1 column 1 char 0) array/mixed-string-table.toml list index out of range array/mixed-int-array.toml Not a homogeneous array (line 1 column 1 char 0) inline-table/multiline.toml Invalid inline table value encountered (line 1 column 1 char 0) inline-table/key-dotted.toml Parsed as unexpected data. 86/100 (86.00%) passed
tomli/tomli_w	string/escape-esc.toml Unescaped '' in a string (at line 1, column 10) 99/100 (99.00%) passed
tomlkit	string/escape-esc.toml Invalid character 'e' in string at line 1 col 8 99/100 (99.00%) passed
rtoml	string/escape-esc.toml invalid escape character in string: `e` at line 1 column 9 99/100 (99.00%) passed
qtoml	datetime/milliseconds.toml Didn't find expected newline (line 2, column 27) datetime/datetime.toml Didn't find expected newline (line 2, column 18) comment/tricky.toml can't parse type (line 11, column 7) string/escape-esc.toml \e not a valid escape (line 1, column 33) string/multiline-quotes.toml Didn't find expected newline (line 4, column 26) 95/100 (95.00%) passed
tomllib	string/escape-esc.toml Unescaped '' in a string (at line 1, column 10) 99/100 (99.00%) passed

Compliance with invalid tests in toml-test

Test the compliance with the standard test suites for invalid toml files here:

https://github.com/BurntSushi/toml-test/

Not OK: The toml file is parsed without error, but expected to fail.
OK: All files are failed to parse, as expected. Showing the last parsing error.

	Result (toml-test v1.3.0)
toml	Not OK: integer/double-sign-plus.toml incorrectly parsed. Not OK: integer/us-after-bin.toml incorrectly parsed. Not OK: integer/double-sign-nex.toml incorrectly parsed. Not OK: integer/us-after-hex.toml incorrectly parsed. Not OK: integer/us-after-oct.toml incorrectly parsed. Not OK: control/comment-del.toml incorrectly parsed. Not OK: control/string-del.toml incorrectly parsed. Not OK: control/multi-us.toml incorrectly parsed. Not OK: control/comment-lf.toml incorrectly parsed. Not OK: control/comment-null.toml incorrectly parsed. Not OK: 39 more items incorrectly parsed. 188/237 (79.32%) passed
tomli/tomli_w	OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18) 237/237 (100%) passed
tomlkit	Not OK: control/comment-cr.toml incorrectly parsed. Not OK: control/bare-cr.toml incorrectly parsed. Not OK: table/append-with-dotted-keys-1.toml incorrectly parsed. Not OK: table/append-with-dotted-keys-2.toml incorrectly parsed. 233/237 (98.31%) passed
rtoml	Not OK: integer/positive-hex.toml incorrectly parsed. Not OK: integer/positive-bin.toml incorrectly parsed. Not OK: control/comment-del.toml incorrectly parsed. Not OK: control/comment-cr.toml incorrectly parsed. Not OK: control/bare-cr.toml incorrectly parsed. 232/237 (97.89%) passed
qtoml	Not OK: control/comment-del.toml incorrectly parsed. Not OK: control/comment-lf.toml incorrectly parsed. Not OK: control/comment-null.toml incorrectly parsed. Not OK: control/comment-cr.toml incorrectly parsed. Not OK: control/comment-us.toml incorrectly parsed. Not OK: control/bare-cr.toml incorrectly parsed. Not OK: table/append-with-dotted-keys-1.toml incorrectly parsed. Not OK: table/duplicate-key-dotted-table.toml incorrectly parsed. Not OK: table/append-with-dotted-keys-2.toml incorrectly parsed. Not OK: table/duplicate-key-dotted-table2.toml incorrectly parsed. Not OK: 2 more items incorrectly parsed. 225/237 (94.94%) passed
tomllib	OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18) 237/237 (100%) passed

Compliance with valid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for valid toml files here:

https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

	Result (cpython tag 3.12.4)
toml	apostrophes-in-literal-string.toml Unbalanced quotes (line 1 column 50 char 49) five-quotes.toml Unterminated string found. Reached end of file. (line 7 column 1 char 97) dates-and-times/datetimes.toml Parsed as unexpected data. multiline-basic-str/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28) 8/12 (66.67%) passed
tomli/tomli_w	OK, 12/12 (100%) passed
tomlkit	OK, 12/12 (100%) passed
rtoml	OK, 12/12 (100%) passed
qtoml	apostrophes-in-literal-string.toml Didn't find expected newline (line 3, column 3) five-quotes.toml Didn't find expected newline (line 3, column 3) dates-and-times/datetimes.toml Didn't find expected newline (line 1, column 19) 9/12 (75.00%) passed
tomllib	OK, 12/12 (100%) passed

Compliance with invalid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for invalid toml files here:

https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid

Not OK: The toml file is parsed without error, but expected to fail.
OK: All files are failed to parse, as expected. Showing the last parsing error.

	Result (cpython tag 3.12.4)
toml	Not OK: invalid-comment-char.toml incorrectly parsed. Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed. Not OK: array/unclosed-empty.toml incorrectly parsed. Not OK: array/file-end-after-val.toml incorrectly parsed. Not OK: array/unclosed-after-item.toml incorrectly parsed. Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed. Not OK: inline-table/unclosed-empty.toml incorrectly parsed. 41/50 (82.00%) passed
tomli/tomli_w	OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21) 50/50 (100%) passed
tomlkit	Not OK: array-of-tables/overwrite-array-in-parent.toml incorrectly parsed. Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-aot.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed. Not OK: inline-table/override-val-in-table.toml incorrectly parsed. 44/50 (88.00%) passed
rtoml	Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed. 49/50 (98.00%) passed
qtoml	Not OK: non-scalar-escaped.toml incorrectly parsed. Not OK: invalid-comment-char.toml incorrectly parsed. Not OK: table/redefine-2.toml incorrectly parsed. Not OK: table/redefine-1.toml incorrectly parsed. Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed. Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed. Not OK: inline-table/override-val-with-table.toml incorrectly parsed. 41/50 (82.00%) passed
tomllib	OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21) 50/50 (100%) passed

Running speed with data provided by `rtoml`

Test the speed of loading and dumping the loaded using data provided by rtoml

https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml

	Loading speed	Dumping speed
toml	Excluded (heterogeneous arrays not supported)	Excluded (heterogeneous arrays not supported)
tomli/tomli_w	2.16s (5000 iterations)	0.74s (5000 iterations)
tomlkit	39.48s (5000 iterations)	1.00s (5000 iterations)
rtoml	0.38s (5000 iterations)	0.09s (5000 iterations)
qtoml	5.00s (5000 iterations)	1.87s (5000 iterations)
tomllib	2.08s (5000 iterations)	Dumping not supported

Running speed with data provided by `tomli`

Test the speed of loading and dumping the loaded using data provided by tomli

https://github.com/hukkin/tomli/raw/master/benchmark/data.toml

	Loading speed	Dumping speed
toml	Excluded (heterogeneous arrays not supported)	Excluded (heterogeneous arrays not supported)
tomli/tomli_w	1.39s (5000 iterations)	0.47s (5000 iterations)
tomlkit	24.42s (5000 iterations)	0.50s (5000 iterations)
rtoml	0.32s (5000 iterations)	0.16s (5000 iterations)
qtoml	3.63s (5000 iterations)	1.25s (5000 iterations)
tomllib	1.38s (5000 iterations)	Dumping not supported

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

with_toml-test_v1.3.0.md

with_toml-test_v1.3.0.md

Report

Version

Dumping `None` value

Dumping key-`None` pair

Dumping list with `None` value

Loading `None`-like values

Dumping a heterogenous array

Loading a heterogenous array

Dumping a nested array

Loading a nested array

Dumping keeps order of keys?

Loading keeps order of keys?

Dumping unicode

Loaded unicode

Compliance with valid tests in toml-test

Compliance with invalid tests in toml-test

Compliance with valid tests in python tomllib test data

Compliance with invalid tests in python tomllib test data

Running speed with data provided by `rtoml`

Running speed with data provided by `tomli`

Files

with_toml-test_v1.3.0.md

Latest commit

History

with_toml-test_v1.3.0.md

File metadata and controls

Report

Version

Dumping None value

Dumping key-None pair

Dumping list with None value

Loading None-like values

Dumping a heterogenous array

Loading a heterogenous array

Dumping a nested array

Loading a nested array

Dumping keeps order of keys?

Loading keeps order of keys?

Dumping unicode

Loaded unicode

Compliance with valid tests in toml-test

Compliance with invalid tests in toml-test

Compliance with valid tests in python tomllib test data

Compliance with invalid tests in python tomllib test data

Running speed with data provided by rtoml

Running speed with data provided by tomli

Dumping `None` value

Dumping key-`None` pair

Dumping list with `None` value

Loading `None`-like values

Running speed with data provided by `rtoml`

Running speed with data provided by `tomli`