JSON backend #338
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| .. _backends-json: | ||
|
|
||
| JSON Backend | ||
| ============ | ||
|
|
||
| openPMD supports writing to and reading from JSON files. | ||
| For this, the installed copy of openPMD must have been built with support for the JSON backend. | ||
| To build openPMD with support for JSON, use the CMake option ``-DopenPMD_USE_JSON=ON``. | ||
| For further information, check out the :ref:`installation guide <install>`, | ||
| :ref:`build dependencies <development-dependencies>` and the :ref:`build options <development-buildoptions>`. | ||
|
|
||
|
|
||
| JSON File Format | ||
| ---------------- | ||
| A JSON file uses the file ending ``.json``. The JSON backend is chosen by creating | ||
| a ``Series`` object with a filename that has this file ending. | ||
|
|
||
| The top-level JSON object is a group representing the openPMD root group ``"/"``. | ||
| Any **openPMD group** is represented in JSON as a JSON object with two reserved keys: | ||
|
|
||
| * ``attributes``: Attributes associated with the group. This key may be null or not be present | ||
| at all, thus indicating a group without attributes. | ||
| * ``platform_byte_widths`` (root group only): Byte widths specific to the writing platform. | ||
| Will be overwritten every time that a JSON value is stored to disk, hence this information | ||
| is only available about the last platform writing the JSON value. | ||
|
|
||
| All datasets and subgroups contained in this group are represented as a further key of | ||
| the group object. ``attributes`` and ``platform_byte_widths`` have | ||
| hence the character of reserved keywords and cannot be used for group and dataset names | ||
| when working with the JSON backend. | ||
| Datasets and groups have the same namespace, meaning that there may not be a subgroup | ||
| and a dataset with the same name contained in one group. | ||
|
|
||
| Any **openPMD dataset** is a JSON object with three keys: | ||
|
|
||
| * ``attributes``: Attributes associated with the dataset. May be ``null`` or not present if no attributes are associated with the dataset. | ||
| * ``datatype``: A string describing the type of the stored data. | ||
| * ``data`` A nested array storing the actual data in row-major manner. | ||
| The data needs to be consistent with the fields ``datatype`` and ``extent``. | ||
| Checking whether this key points to an array can be (and is internally) used to distinguish groups from datasets. | ||
|
|
||
| **Attributes** are stored as a JSON object with a key for each attribute. | ||
| Every such attribute is itself a JSON object with two keys: | ||
|
|
||
| * ``datatype``: A string describing the type of the value. | ||
| * ``value``: The actual value of type ``datatype``. | ||
|
|
||
| Restrictions | ||
| ------------ | ||
| For creation of JSON serializations (i.e. writing), the restrictions of the JSON backend are | ||
| equivalent to those of the `JSON library by Niels Lohmann <https://github.com/nlohmann/json>`_ | ||
| used by the openPMD backend. | ||
|
|
||
| Numerical values, integral as well as floating point, are supported up to a length of | ||
| 64 bits. | ||
| Since JSON does not support special floating point values (i.e. NaN, Infinity, -Infinity), | ||
| those values are rendered as ``null``. | ||
|
|
||
| Instructing openPMD to write values of a datatype that is too wide for the JSON | ||
| backend does *not* result in an error: | ||
| * If casting the value to the widest supported datatype of the same category (integer or floating point) | ||
| is possible without data loss, the cast is performed and the value is written. | ||
| As an example, on a platform with ``sizeof(double) == 8``, writing the value | ||
| ``static_cast<long double>(std::numeric_limits<double>::max())`` will work as expected | ||
| since it can be cast back to ``double``. | ||
| * Otherwise, a ``null`` value is written. | ||
|
|
||
| Upon reading ``null`` when expecting a floating point number, a NaN value will be | ||
| returned. Take notice that a NaN value returned from the deserialization process | ||
| may have originally been +/-Infinity or beyond the supported value range. | ||
|
|
||
| Upon reading ``null`` when expecting any other datatype, the JSON backend will | ||
| propagate the exception thrown by Niels Lohmann's library. | ||
|
|
||
| A parallel (i.e. MPI) implementation is *not* available. | ||
|
|
||
| Example | ||
| ------- | ||
| The example code in the :ref:`usage section <usage-serial>` will produce the following JSON serialization | ||
| when picking the JSON backend: | ||
|
|
||
| .. literalinclude:: json_example.json | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| { | ||
|
There was a problem hiding this comment. When we are finished with the review (until then, pls update here): |
||
| "attributes": { | ||
| "basePath": { | ||
| "datatype": "STRING", | ||
| "value": "/data/%T/" | ||
| }, | ||
| "iterationEncoding": { | ||
| "datatype": "STRING", | ||
| "value": "groupBased" | ||
| }, | ||
| "iterationFormat": { | ||
| "datatype": "STRING", | ||
| "value": "/data/%T/" | ||
| }, | ||
| "meshesPath": { | ||
| "datatype": "STRING", | ||
| "value": "meshes/" | ||
| }, | ||
| "openPMD": { | ||
| "datatype": "STRING", | ||
| "value": "1.1.0" | ||
| }, | ||
| "openPMDextension": { | ||
| "datatype": "UINT", | ||
| "value": 0 | ||
| } | ||
| }, | ||
| "data": { | ||
| "1": { | ||
| "attributes": { | ||
| "dt": { | ||
| "datatype": "DOUBLE", | ||
| "value": 1 | ||
| }, | ||
| "time": { | ||
| "datatype": "DOUBLE", | ||
| "value": 0 | ||
| }, | ||
| "timeUnitSI": { | ||
| "datatype": "DOUBLE", | ||
| "value": 1 | ||
| } | ||
| }, | ||
| "meshes": { | ||
| "rho": { | ||
| "attributes": { | ||
| "axisLabels": { | ||
| "datatype": "VEC_STRING", | ||
| "value": [ | ||
| "x" | ||
| ] | ||
| }, | ||
| "dataOrder": { | ||
| "datatype": "STRING", | ||
| "value": "C" | ||
| }, | ||
| "geometry": { | ||
| "datatype": "STRING", | ||
| "value": "cartesian" | ||
| }, | ||
| "gridGlobalOffset": { | ||
| "datatype": "VEC_DOUBLE", | ||
| "value": [ | ||
| 0 | ||
| ] | ||
| }, | ||
| "gridSpacing": { | ||
| "datatype": "VEC_DOUBLE", | ||
| "value": [ | ||
| 1 | ||
| ] | ||
| }, | ||
| "gridUnitSI": { | ||
| "datatype": "DOUBLE", | ||
| "value": 1 | ||
| }, | ||
| "position": { | ||
| "datatype": "VEC_DOUBLE", | ||
| "value": [ | ||
| 0 | ||
| ] | ||
| }, | ||
| "timeOffset": { | ||
| "datatype": "FLOAT", | ||
| "value": 0 | ||
| }, | ||
| "unitDimension": { | ||
| "datatype": "ARR_DBL_7", | ||
| "value": [ | ||
| 0, | ||
| 0, | ||
| 0, | ||
| 0, | ||
| 0, | ||
| 0, | ||
| 0 | ||
| ] | ||
| }, | ||
| "unitSI": { | ||
| "datatype": "DOUBLE", | ||
| "value": 1 | ||
| } | ||
| }, | ||
| "data": [ | ||
| [ | ||
| 0, | ||
| 1, | ||
| 2 | ||
| ], | ||
| [ | ||
| 3, | ||
| 4, | ||
| 5 | ||
| ], | ||
| [ | ||
| 6, | ||
| 7, | ||
| 8 | ||
| ] | ||
| ], | ||
| "datatype": "DOUBLE" | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "platform_byte_widths": { | ||
| "BOOL": 1, | ||
| "CHAR": 1, | ||
| "DOUBLE": 8, | ||
| "FLOAT": 4, | ||
| "INT": 4, | ||
| "LONG": 8, | ||
| "LONGLONG": 8, | ||
| "LONG_DOUBLE": 16, | ||
| "SHORT": 2, | ||
| "UCHAR": 1, | ||
| "UINT": 4, | ||
| "ULONG": 8, | ||
| "ULONGLONG": 8, | ||
| "USHORT": 2 | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| .. _usage-firststeps: | ||
|
|
||
| First Steps | ||
| =========== | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| .. _usage-parallel: | ||
|
|
||
| Parallel API | ||
| ============ | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| .. _usage-serial: | ||
|
|
||
| Serial API | ||
| ========== | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,7 @@ enum class Format | |
| HDF5, | ||
| ADIOS1, | ||
| ADIOS2, | ||
| JSON, | ||
| DUMMY | ||
| }; //Format | ||
| } // openPMD | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe link to https://github.com/openPMD/openPMD-example-datasets when it's included there?