JSON backend

Implement JSON backend to dummy state Misses the actual implementation of AbstractIOHandlerImpl Declare IOHandlerImpl for JSON and integrate it with other places Misses the implementation. Undebugged minimum implementation for JSON writing First basically runnable version of JSON writing To address: No reading or deleting yet. Datatypes are currently ignored and the data is assumed to be int64_t. Attribute values are ignored and replaced with a dummy value. If a subgroup name can be parsed as a nonnegative string, the JSON API will create a JSON array rather than a JSON object (associative array) as intended. Correctly handle groups that can be parsed as int See last commit's description. Fix index calculation with offsets in WriteData Fix some mistakes in JSON writing Correctly handle overwriting files: -> overwritten files should not be possible to access any longer and be clearly distinguished from the newly-created file Make some verifications execute independent of compiler options. Full implementation of JSON writing Respects all datatypes now. Format code according to Clion Stylesheet https://github.com/ComputationalRadiationPhysics/contributing/blob/master/IDESettings/CLion/CRP_CLion2016_1.xml Add generic branching over an openPMD datatype First runnable version of JSON Reading Cleanup and implementation of dataset extension Undebugged version of JSON deletion Properly (de)serialize datatypes Instead of casting the Datatype enum to and from int (which is likely to break when altering the enum), serialize to and from String values. Fix a number of mistakes in JSON reading and writing Cleanup Add JSON tests and fix bugs found thusly Add further tests and fix a further bug The JSON library does not understand long double values (i.e. 128bit floats), represent them as a char array. Handle floating point special values JSON represents +/-Infinity and NaN values as null. The JSON library will correctly serialize those values *to* JSON, implement (semi)-correct handly for deserialization. As it is unclear which exact value a null represents, deserialize it to NaN. Take notice that large floating point values (128 bit) might be serialized to null as well. Use std::is_floating_point to distinguish them from other types Additionally write the byte width of the underlying type Not yet used in reading Mark the writable written after successfully extending a dataset Remove support for absolute paths from openPath Fix some rough edges from rebasing Add documentation for the JSON backend Integrate the JSON backend with the build system Make platform bytewidth information global per JSON file Was previously annotated for every single Dataset and Attribute. Fixes -> shadowed variables -> return file streams by shared pointer so that Clang won't be mad about missing copy constructors -> handle inconsistencies between GCC, Clang and MSVC in switchType function Add licensing information and cleanup imports Cleanup Remove unused code from Filesystem.(h|c)pp Unify replace_all and replace_all_nonrecursively in StringManip.hpp Move implementation of JSONFilePosition class to source file Formatting Refactor tests Move tests into functions of the backend to be used. Remove /subgroups/ and /datasets/ indirection Make optional_paths_110_test HDF5 specific again Eliminate "extent" field from datasets Remove object_type key
openPMD · Dec 12, 2018 · 0999600 · 0999600
1 parent 8980b71
commit 0999600
Show file tree

Hide file tree

Showing 23 changed files with 4,049 additions and 1,668 deletions.
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -304,7 +304,10 @@ set(IO_SOURCE
         src/IO/AbstractIOHandlerHelper.cpp
         src/IO/IOTask.cpp
         src/IO/HDF5/HDF5IOHandler.cpp
-        src/IO/HDF5/ParallelHDF5IOHandler.cpp)
+        src/IO/HDF5/ParallelHDF5IOHandler.cpp
+        src/IO/JSON/JSONIOHandler.cpp
+        src/IO/JSON/JSONIOHandlerImpl.cpp
+        src/IO/JSON/JSONFilePosition.cpp)
 set(IO_ADIOS1_SEQUENTIAL_SOURCE
         src/IO/AbstractIOHandler.cpp
         src/IO/AbstractIOHandlerImpl.cpp

diff --git a/README.md b/README.md
@@ -100,7 +100,7 @@ Shipped internally in `share/openPMD/thirdParty/`:
 * [NLohmann-JSON](https://github.com/nlohmann/json) 3.4.0+ ([MIT](https://github.com/nlohmann/json/blob/develop/LICENSE.MIT))
 
 Optional I/O backends:
-* [JSON](https://en.wikipedia.org/wiki/JSON) (*not yet implemented*)
+* [JSON](https://en.wikipedia.org/wiki/JSON)
 * [HDF5](https://support.hdfgroup.org/HDF5) 1.8.13+
 * [ADIOS1](https://www.olcf.ornl.gov/center-projects/adios) 1.13.1+
 * [ADIOS2](https://github.com/ornladios/ADIOS2) 2.1+ (*not yet implemented*)
@@ -175,7 +175,7 @@ CMake controls options with prefixed `-D`, e.g. `-DopenPMD_USE_MPI=OFF`:
 | CMake Option                 | Values           | Description                                                                  |
 |------------------------------|------------------|------------------------------------------------------------------------------|
 | `openPMD_USE_MPI`            | **AUTO**/ON/OFF  | Enable MPI support                                                           |
-| `openPMD_USE_JSON`           | **AUTO**/ON/OFF  | Enable support for JSON <sup>1</sup>                                         |
+| `openPMD_USE_JSON`           | **AUTO**/ON/OFF  | Enable support for JSON                                                      |
 | `openPMD_USE_HDF5`           | **AUTO**/ON/OFF  | Enable support for HDF5                                                      |
 | `openPMD_USE_ADIOS1`         | **AUTO**/ON/OFF  | Enable support for ADIOS1                                                    |
 | `openPMD_USE_ADIOS2`         | AUTO/ON/**OFF**  | Enable support for ADIOS2 <sup>1</sup>                                       |

diff --git a/docs/source/backends/json.rst b/docs/source/backends/json.rst
@@ -0,0 +1,83 @@
+.. _backends-json:
+
+JSON Backend
+============
+
+openPMD supports writing to and reading from JSON files.
+For this, the installed copy of openPMD must have been built with support for the JSON backend.
+To build openPMD with support for JSON, use the CMake option ``-DopenPMD_USE_JSON=ON``.
+For further information, check out the :ref:`installation guide <install>`,
+:ref:`build dependencies <development-dependencies>` and the :ref:`build options <development-buildoptions>`.
+
+
+JSON File Format
+----------------
+A JSON file uses the file ending ``.json``. The JSON backend is chosen by creating
+a ``Series`` object with a filename that has this file ending.
+
+The top-level JSON object is a group representing the openPMD root group ``"/"``.
+Any **openPMD group** is represented in JSON as a JSON object with two reserved keys:
+
+ * ``attributes``: Attributes associated with the group. This key may be null or not be present
+   at all, thus indicating a group without attributes.
+ * ``platform_byte_widths`` (root group only): Byte widths specific to the writing platform.
+   Will be overwritten every time that a JSON value is stored to disk, hence this information 
+   is only available about the last platform writing the JSON value.
+
+All datasets and subgroups contained in this group are represented as a further key of
+the group object. ``attributes`` and ``platform_byte_widths`` have
+hence the character of reserved keywords and cannot be used for group and dataset names
+when working with the JSON backend.
+Datasets and groups have the same namespace, meaning that there may not be a subgroup
+and a dataset with the same name contained in one group.
+
+Any **openPMD dataset** is a JSON object with three keys:
+
+ * ``attributes``: Attributes associated with the dataset. May be ``null`` or not present if no attributes are associated with the dataset.
+ * ``datatype``: A string describing the type of the stored data.
+ * ``data`` A nested array storing the actual data in row-major manner.
+   The data needs to be consistent with the fields ``datatype`` and ``extent``.
+   Checking whether this key points to an array can be (and is internally) used to distinguish groups from datasets.
+
+**Attributes** are stored as a JSON object with a key for each attribute.
+Every such attribute is itself a JSON object with two keys:
+
+ * ``datatype``: A string describing the type of the value.
+ * ``value``: The actual value of type ``datatype``.
+
+Restrictions
+------------
+For creation of JSON serializations (i.e. writing), the restrictions of the JSON backend are
+equivalent to those of the `JSON library by Niels Lohmann <https://github.com/nlohmann/json>`_
+used by the openPMD backend.
+
+Numerical values, integral as well as floating point, are supported up to a length of
+64 bits.
+Since JSON does not support special floating point values (i.e. NaN, Infinity, -Infinity),
+those values are rendered as ``null``.
+
+Instructing openPMD to write values of a datatype that is too wide for the JSON
+backend does *not* result in an error:
+ * If casting the value to the widest supported datatype of the same category (integer or floating point)
+   is possible without data loss, the cast is performed and the value is written.
+   As an example, on a platform with ``sizeof(double) == 8``, writing the value
+   ``static_cast<long double>(std::numeric_limits<double>::max())`` will work as expected
+   since it can be cast back to ``double``.
+ * Otherwise, a ``null`` value is written.
+
+Upon reading ``null`` when expecting a floating point number, a NaN value will be
+returned. Take notice that a NaN value returned from the deserialization process
+may have originally been +/-Infinity or beyond the supported value range.
+
+Upon reading ``null`` when expecting any other datatype, the JSON backend will
+propagate the exception thrown by Niels Lohmann's library.
+
+A parallel (i.e. MPI) implementation is *not* available.
+
+Example
+-------
+The example code in the :ref:`usage section <usage-serial>` will produce the following JSON serialization
+when picking the JSON backend:
+
+.. literalinclude:: json_example.json
+
diff --git a/docs/source/backends/json_example.json b/docs/source/backends/json_example.json
@@ -0,0 +1,142 @@
+{
+  "attributes": {
+    "basePath": {
+      "datatype": "STRING",
+      "value": "/data/%T/"
+    },
+    "iterationEncoding": {
+      "datatype": "STRING",
+      "value": "groupBased"
+    },
+    "iterationFormat": {
+      "datatype": "STRING",
+      "value": "/data/%T/"
+    },
+    "meshesPath": {
+      "datatype": "STRING",
+      "value": "meshes/"
+    },
+    "openPMD": {
+      "datatype": "STRING",
+      "value": "1.1.0"
+    },
+    "openPMDextension": {
+      "datatype": "UINT",
+      "value": 0
+    }
+  },
+  "data": {
+    "1": {
+      "attributes": {
+        "dt": {
+          "datatype": "DOUBLE",
+          "value": 1
+        },
+        "time": {
+          "datatype": "DOUBLE",
+          "value": 0
+        },
+        "timeUnitSI": {
+          "datatype": "DOUBLE",
+          "value": 1
+        }
+      },
+      "meshes": {
+        "rho": {
+          "attributes": {
+            "axisLabels": {
+              "datatype": "VEC_STRING",
+              "value": [
+                "x"
+              ]
+            },
+            "dataOrder": {
+              "datatype": "STRING",
+              "value": "C"
+            },
+            "geometry": {
+              "datatype": "STRING",
+              "value": "cartesian"
+            },
+            "gridGlobalOffset": {
+              "datatype": "VEC_DOUBLE",
+              "value": [
+                0
+              ]
+            },
+            "gridSpacing": {
+              "datatype": "VEC_DOUBLE",
+              "value": [
+                1
+              ]
+            },
+            "gridUnitSI": {
+              "datatype": "DOUBLE",
+              "value": 1
+            },
+            "position": {
+              "datatype": "VEC_DOUBLE",
+              "value": [
+                0
+              ]
+            },
+            "timeOffset": {
+              "datatype": "FLOAT",
+              "value": 0
+            },
+            "unitDimension": {
+              "datatype": "ARR_DBL_7",
+              "value": [
+                0,
+                0,
+                0,
+                0,
+                0,
+                0,
+                0
+              ]
+            },
+            "unitSI": {
+              "datatype": "DOUBLE",
+              "value": 1
+            }
+          },
+          "data": [
+            [
+              0,
+              1,
+              2
+            ],
+            [
+              3,
+              4,
+              5
+            ],
+            [
+              6,
+              7,
+              8
+            ]
+          ],
+          "datatype": "DOUBLE"
+        }
+      }
+    }
+  },
+  "platform_byte_widths": {
+    "BOOL": 1,
+    "CHAR": 1,
+    "DOUBLE": 8,
+    "FLOAT": 4,
+    "INT": 4,
+    "LONG": 8,
+    "LONGLONG": 8,
+    "LONG_DOUBLE": 16,
+    "SHORT": 2,
+    "UCHAR": 1,
+    "UINT": 4,
+    "ULONG": 8,
+    "ULONGLONG": 8,
+    "USHORT": 2
+  }
+}
diff --git a/docs/source/dev/buildoptions.rst b/docs/source/dev/buildoptions.rst
@@ -15,7 +15,7 @@ CMake controls options with prefixed ``-D``, e.g. ``-DopenPMD_USE_MPI=OFF``:
 CMake Option                   Values          Description
 ============================== =============== ========================================================================
 ``openPMD_USE_MPI``            **AUTO**/ON/OFF Enable MPI support
-``openPMD_USE_JSON``           **AUTO**/ON/OFF Enable support for JSON :sup:`1`
+``openPMD_USE_JSON``           **AUTO**/ON/OFF Enable support for JSON
 ``openPMD_USE_HDF5``           **AUTO**/ON/OFF Enable support for HDF5
 ``openPMD_USE_ADIOS1``         **AUTO**/ON/OFF Enable support for ADIOS1
 ``openPMD_USE_ADIOS2``         AUTO/ON/**OFF** Enable support for ADIOS2 :sup:`1`

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -84,6 +84,17 @@ Development
    dev/doxygen
    dev/release
 
+********
+Backends
+********
+.. toctree::
+   :caption: BACKENDS
+   :maxdepth: 1
+   :hidden:
+
+   backends/json
+
+
 *********
 Utilities
 *********
@@ -93,3 +104,4 @@ Utilities
    :hidden:
 
    utilities/benchmark.rst
+
diff --git a/docs/source/usage/firststeps.rst b/docs/source/usage/firststeps.rst
@@ -1,4 +1,4 @@
-.. usage-firststeps:
+.. _usage-firststeps:
 
 First Steps
 ===========

diff --git a/docs/source/usage/parallel.rst b/docs/source/usage/parallel.rst
@@ -1,4 +1,4 @@
-.. usage-parallel:
+.. _usage-parallel:
 
 Parallel API
 ============

diff --git a/docs/source/usage/serial.rst b/docs/source/usage/serial.rst
@@ -1,4 +1,4 @@
-.. usage-serial:
+.. _usage-serial:
 
 Serial API
 ==========

diff --git a/include/openPMD/Datatype.hpp b/include/openPMD/Datatype.hpp
@@ -519,6 +519,11 @@ isSame( openPMD::Datatype const d, openPMD::Datatype const e )
     return false;
 }
 
+#if _MSC_VER && !__INTEL_COMPILER
+#define OPENPMD_TEMPLATE_OPERATOR operator
+#else
+#define OPENPMD_TEMPLATE_OPERATOR template operator
+#endif
 /**
  * Generalizes switching over an openPMD datatype.
  *
@@ -536,12 +541,6 @@ isSame( openPMD::Datatype const d, openPMD::Datatype const e )
  * the passed arguments and the template parameter type corresponding to the
  * openPMD type.
  */
-
-#if _MSC_VER && !__INTEL_COMPILER
-#define OPENPMD_TEMPLATE_OPERATOR operator
-#else
-#define OPENPMD_TEMPLATE_OPERATOR template operator
-#endif
 template<
     typename ReturnType = void,
     typename Action,

diff --git a/include/openPMD/IO/Format.hpp b/include/openPMD/IO/Format.hpp
@@ -30,6 +30,7 @@ enum class Format
     HDF5,
     ADIOS1,
     ADIOS2,
+    JSON,
     DUMMY
 };  //Format
 } // openPMD
diff --git a/include/openPMD/IO/JSON/JSONFilePosition.hpp b/include/openPMD/IO/JSON/JSONFilePosition.hpp
@@ -0,0 +1,49 @@
+/* Copyright 2017-2018 Franz Pöschel
+ *
+ * This file is part of openPMD-api.
+ *
+ * openPMD-api is free software: you can redistribute it and/or modify
+ * it under the terms of of either the GNU General Public License or
+ * the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * openPMD-api is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License and the GNU Lesser General Public License
+ * for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * and the GNU Lesser General Public License along with openPMD-api.
+ * If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#pragma once
+
+
+#include "openPMD/IO/AbstractFilePosition.hpp"
+
+
+#if openPMD_HAVE_JSON
+#include <nlohmann/json.hpp>
+#endif
+
+namespace openPMD
+{
+
+
+    struct JSONFilePosition :
+        public AbstractFilePosition
+#if openPMD_HAVE_JSON
+    {
+        using json = nlohmann::json;
+        json::json_pointer id;
+
+        JSONFilePosition( json::json_pointer ptr = json::json_pointer( ) );
+    };
+#else
+    {};
+#endif
+
+} // openPMD
-Original file line number
+Diff line change
@@ Expand Up / @@ -30,6 +30,7 @@ enum class Format @@
         HDF5,
         ADIOS1,
         ADIOS2,
+        JSON,
         DUMMY
     };  //Format
     } // openPMD