diff --git a/docs/how-tos/attachments/51dc82f9f0f5ec2f-image.png b/docs/how-tos/attachments/51dc82f9f0f5ec2f-image.png new file mode 100644 index 000000000..32e0d3300 Binary files /dev/null and b/docs/how-tos/attachments/51dc82f9f0f5ec2f-image.png differ diff --git a/docs/how-tos/attachments/5cbd8c6a1ca227df-image.png b/docs/how-tos/attachments/5cbd8c6a1ca227df-image.png new file mode 100644 index 000000000..02da891af Binary files /dev/null and b/docs/how-tos/attachments/5cbd8c6a1ca227df-image.png differ diff --git a/docs/how-tos/attachments/9d748230d54a8059-image.png b/docs/how-tos/attachments/9d748230d54a8059-image.png new file mode 100644 index 000000000..762c21cb9 Binary files /dev/null and b/docs/how-tos/attachments/9d748230d54a8059-image.png differ diff --git a/docs/how-tos/attachments/c6ab2f4b925aed27-image.png b/docs/how-tos/attachments/c6ab2f4b925aed27-image.png new file mode 100644 index 000000000..c86ea8bd2 Binary files /dev/null and b/docs/how-tos/attachments/c6ab2f4b925aed27-image.png differ diff --git a/docs/how-tos/attachments/d8e727b3b32dcbb9-image.png b/docs/how-tos/attachments/d8e727b3b32dcbb9-image.png new file mode 100644 index 000000000..2dd0cffaa Binary files /dev/null and b/docs/how-tos/attachments/d8e727b3b32dcbb9-image.png differ diff --git a/docs/how-tos/attachments/e28ec15bb537c9b7-image.png b/docs/how-tos/attachments/e28ec15bb537c9b7-image.png new file mode 100644 index 000000000..ebbe36ec4 Binary files /dev/null and b/docs/how-tos/attachments/e28ec15bb537c9b7-image.png differ diff --git a/docs/how-tos/create-nexus-files-by-python.md b/docs/how-tos/create-nexus-files-by-python.md new file mode 100644 index 000000000..2aa993bb2 --- /dev/null +++ b/docs/how-tos/create-nexus-files-by-python.md @@ -0,0 +1,246 @@ +# Use python to create NeXus files + +__The goal__ + +Use python to create a NeXus file (.nxs) by hardcoding via the python package `h5py`. NeXus files can also be created by our software [`pynxtools`](https://github.com/FAIRmat-NFDI/pynxtools) automatically, but ONLY IF a reader for the specific device/instrument/data-structure exists. This How-To is intended as easy access to FAIR data structures _via_ NeXus. For static-datastructures (i.e., always the same type of standard measurement) or one-time examples (small data publications), this may provide a feasible solution. For large scaled automated file processing, storage, and validation, it is advisable to use [`pynxtools`](https://github.com/FAIRmat-NFDI/pynxtools) and its measurement method specific [plugins](../reference/plugins.md) + +You can find the necessary file downloads [here](https://zenodo.org/records/13373909). + + + +## Make NeXus file by python + +Install `h5py` via `pip`: +```console +`pip install h5py` +``` + +Then you can create a NeXus file by the python script called [h5py_nexus_file_creation.py](https://zenodo.org/records/13373909/files/h5py_nexus_file_creation.py?download=1). + +``` +# Import h5py, to write an hdf5 file +import h5py + +# create a h5py file in writing mode with given name "NXopt_minimal_example", file extension "nxs" +f = h5py.File("NXopt_minimal_example.nxs", "w") + +# there are only 3 fundamental objects: >group<, >attribute< and >datafield<. + + +# create a >group< called "entry" +f.create_group('/entry') + +# assign the >group< called "entry" an >attribute< +# The attribute is "NX_class"(a NeXus class) with the value of this class is "NXentry" +f['/entry'].attrs['NX_class'] = 'NXentry' + +# create >datafield< called "definition" inside the entry, and assign it the value "NXoptical_spectroscopy" +# This field is important, as it is used in validation process to identify the NeXus definition. +f['/entry/definition'] = 'NXoptical_spectroscopy' +``` + +This proves a starting point of the NeXus file. We will go through these functions in the following. + + + +## Add NeXus concepts by python + +Go to [FAIRmat NeXus definitions]() + +Scroll down until you see the search box named "Quick search". + +Type "NXoptical" and press start the search. + +You see several search results, select the one with is named "NXoptical\_spectroscopy". + +Then you are (ideally) on this page: [NXoptical_spectroscopy NeXus definition]() + +You see a tree-like structure of the NeXus definition NXoptical\_spectrosocopy with several tree nodes: Status, Description, Symbols, Groups\_cited, Structure. For now, only the part in Structure is of interest. This contains the information which has to be written in the python code to add fields/groups/attributes to the NeXus file. + +Use your browser search (CRTL+F) and search for "required". Ideally, your browser highlights all concepts which are required. You have to add those to the python script to extend your created .nxs file. (Which fields/groups/attributes are "required" was defined by the respective scientific community, to ensure that the data serves the FAIR principles.) + +In the following, it will be shown how the python script has to be extended for the three fundamental objects: + +1. Attribute + +2. Datafield + +3. Group + + + + + +### Adding an attribute + +Search for the first concept/object in the NeXus file which is not created yet. It is: + +**@version**: (required) [NX\_CHAR]() [⤆]() + +1. It is located in the tree at position: ENTRY/definition/ + +2. The "@" indicates that this is an attribute of the concept "definition". + +3. The name of the attribute is "version". + +4. Since it is "required", that means this attribute has to be added so that the resulting NeXus file is compliant with the NeXus definition "NXoptical\_spectroscopy". + +5. The "NX\_CHAR" indicates the datatype. This should be a string: "The preferred string representation is UTF-8" (more information see [here]()) + +![image.png](<./attachments/51dc82f9f0f5ec2f-image.png>) + +Now the python script has to be extended in the following: + +``` +f['/entry/definition'].attrs['version'] = 'v2024.02' +``` + +This h5py command adds the attribute named "version" with the value "v2024.02" to the HDF5 dataset called "/entry/definition". The same is done for the URL attribute: + +``` +f['/entry/definition'].attrs['URL'] = 'https://github.com/FAIRmat-NFDI/nexus_definitions/blob/f75a29836431f35d68df6174e3868a0418523397/contributed_definitions/NXoptical_spectroscopy.nxdl.xml' +``` + +For your use case, you may want to use a different version of the NeXus definitions, since these are changed over time. In the following, it is shown where to obtain the correct version and URL. + +__Get the values: *version* and *URL*__ + +At the time, you create the NeXus definition. Go to the page of the respectively used NeXus concept, i.e. [NXoptical_spectroscopy]() + +Scroll down until you find "**NXDL Source**:" and follow this link, i.e. [NXoptical_spectroscopy.nxdl.xml]() + +This is the GitHub website, in which the latest (FAIRmat) NeXus definition of NXoptical\_spectroscopy is stored in the NeXus definition language file (.nxdl). The information is structured in the xml format. + +Now you have to copy the permalink of this file. Go to the top right side of the website. Find the Menu made by 3 dots: + +![image.png](<./attachments/c6ab2f4b925aed27-image.png>) + +Copy the permalink and insert it as value for the "URL" attribute (Step 1, Red box in the image) + +Go to "nexus\_definitions" (Step 2, Red box in the image) + +![image.png](<./attachments/d8e727b3b32dcbb9-image.png>) + +On the right side, you should see below "Releases" the "tags" (Red box in the image). Follow this link. + +Copy the latest tag, which should look similar to "v2024.02". Insert it as value for the "version" attribute. + +__Disclaimer__ +When specifying this version tag, it would be better to include the "GitHub commit id" as well. In this way, a [pynxtools generated version tag](https://github.com/FAIRmat-NFDI/pynxtools/blob/c13716915bf8f69068c3b94d1423681b580fd437/src/pynxtools/_build_wrapper.py#L17) might look like this: +`v2022.07.post1.dev1278+g1d7000f4`. If you have pynxtools installed, you can get the tag by: + +```python +>>> from pynxtools import get_nexus_version +>>> get_nexus_version() +'v2022.07.post1.dev1284+gf75a2983' +``` + + + +### Adding a datafield + +Two attributes were added to "ENTRY/definition", both of which were required. By now, this part of the NeXus file fulfills the requirements of the application definition NXoptical\_spectroscopy. + +The next required concept of [NXoptical_spectrsocopy](https://fairmat-nfdi.github.io/nexus_definitions/classes/contributed_definitions/NXoptical_spectroscopy.html) is "**experiment\_type"**. + +**experiment\_type**: (required) [NX\_CHAR]() + +1. It is located in the tree at position: ENTRY/ + +2. There is no "@" in front of "**experiment\_type"**. So, this may be a group or a datafield. + +3. The name of this group/datafield is "**experiment\_type**". + +4. The "required" indicates that this group/datafield has to be added to be in line with the NeXus definition "NXoptical\_spectroscopy". + +5. The "NX\_CHAR" indicates the datatype. This should be a string: "The preferred string representation is UTF-8" (more information see [here]()). + +6. The "NX\_CHAR" indicates that this is a datafield. It is NOT a group. + A group is a NeXus class. "NXentry" is for example a NeXus class, while "NX_CHAR" indicates the datatype of the field. + Whether or not the underscore "_" is present after NX, indicates therefore if it is a NeXus class or datafield. + +Read the documentation at "▶ Specify the type of the optical experiment. ..." by extending it via click on the triangle symbol. You should see something like this: + +![image.png](<./attachments/5cbd8c6a1ca227df-image.png>) + +There, the value of the datafield has to be one of the shown list, since it is an enumeration (e.g. "transmission spectroscopy"). Note that this is case sensitive. + +Therefore, the python script has to be extended by: + +``` +f['/entry/experiment_type'] = 'transmission spectroscopy' +``` + + + + + +### Adding a group + +The first required group in NXoptical\_spectroscopy on the "ENTRY/" level is "**INSTRUMENT**: (required) [NXinstrument]() [⤆"]() + +1. It is located in the tree at position: NXentry/ + +2. There is no "@" in front of "**INSTRUMENT"** and because the "NXinstrument" is a NeXus class, this has to be implemented as group in the python script. + +3. The "required" indicates that this group has to be added to be in line with the NeXus definition "NXoptical\_spectroscopy". + +4. The "NXinstrument" indicates that it is a NeXus class (or group in python), as it starts with "NX" - without an underscore "_". It can also not be found at the [data types](https://manual.nexusformat.org/nxdl-types.html#data-types-allowed-in-nxdl-specifications). + +5. As this is a group, attributes or values may be assigned to it. + +6. As this is a group, it can contain many datafields or groups. + +7. The uppercase notation of "**INSTRUMENT**" means: + + 1. You can give INSTRUMENT [almost](https://manual.nexusformat.org/datarules.html) any name, such as "abc" or "Raman\_setup" (see "regex" or regular expression). + + 2. You can create as many groups with the class NXinstrument as you want. Their names have to be different. + + 3. For more information see the [NeXus rules](../learn/nexus-rules.md) + +The respective python code to implement a NXinstrument class (or equivalently in python group) with the name "experiment\_setup\_1" is: + +``` +f.create_group('/entry/experiment_setup_1') +f['/entry/experiment_setup_1'].attrs['NX_class'] = 'NXinstrument' +``` + +The first line creates the group with the name "experiment\_setup\_1". + +The second line assigns this group the attribute with the name "NX\_class" and its value "NXinstrument". + + + + + +### Finishing the NeXus file + +This has to be done by using the respective NeXus definition website: + +[NXoptical_spectroscopy]() + +And by searching for all "required" entries. The next required entries are located inside the NXinstrument class: + +1. **beam\_TYPE**: (required) [NXbeam]() [⤆]() + +2. **detector\_TYPE**: (required) [NXdetector]() [⤆]() + +Both are groups. "**beam\_TYPE"** could be named: "beam\_abc" or "beam\_Raman\_setup". Use the knowledge above to extend the python script to create those NeXus file entries. + +__Note for required NeXus concepts__ + +Above in the definition of NXoptical\_spectroscopy, you as well may found a required entry "**depends\_on**: (required) [NX\_CHAR]() [⤆"](). This is at the level of "ENTRY/reference\_frames/beam\_ref\_frame". If you don't have the group "**beam\_ref\_frame"** because this is "optional", then you don't need to have this field. + + + +[_Continue by validating the NeXus file_](validate-nexus-file.md) + +## Feedback and contact + +1. Best way is to contact the FAIRmat team directly by creating a [Github Issue](https://github.com/FAIRmat-NFDI/nexus_definitions/issues/new). + +2. ron.hildebrandt(at)physik.hu-berlin.de + + + diff --git a/docs/how-tos/installation_notes_nxvalidate.md b/docs/how-tos/installation_notes_nxvalidate.md new file mode 100644 index 000000000..7c725ea41 --- /dev/null +++ b/docs/how-tos/installation_notes_nxvalidate.md @@ -0,0 +1,80 @@ + +This lists some notes for installation of nxvalidate on Ubuntu and Windows. For windows, the installation of the XML2 library was not sucessful. This should be possible, but could not reproduced yet. + + + +# cnxvalidate installation on Ubuntu 22.04 + +These commands install nxvaldiate on a fresh Ubuntu 22.04 system (tested with Linux running from USB stick). + +``` +sudo apt-get update +sudo apt-get install git +sudo apt-get install build-essential +sudo add-apt-repository universe +sudo apt-get install libhdf5-serial-dev +sudo apt-get -y install pkg-config +sudo apt upgrade -y +sudo apt-get -y install cmake +sudo apt-get install libxml2-dev + +mkdir nexusvalidate +cd nexusvalidate +git clone https://github.com/nexusformat/cnxvalidate.git +cd cnxvalidate/ +mkdir build +cd build/ +cmake ../ +make +``` + +# cnxvalidate installation on windows: + +## -- CMAKE + +[https://cmake.org/download/]() + +\--> [cmake-3.30.2-windows-x86\_64.msi]() + +Install with .msi + +## -- HDF5 + +Download **hdf5-1.14.4-2-win-vs2022\_**[**cl.zip**]()** from **[https://www.hdfgroup.org/downloads/hdf5/]() + +unzip the .zip file + +put the file into the folder + +``` +C:\hdf5 +``` + +(can be named differently, but no spaces are allowed for this path) + +``` +set PATH=%PATH%;C:\your\path\here\ +``` + +## -- libiconv + +[https://github.com/vovythevov/libiconv-cmake]() + +``` +git clone +``` + +cd to downloaded directory + +``` +mkdir build +cd build +cmake .. +``` + +## -- XML2 + +??? Unsolved... + +Please create GitHub issue [here](https://github.com/FAIRmat-NFDI/pynxtools/issues/new) if you could solve this. + diff --git a/docs/how-tos/testing-validation-tools.md b/docs/how-tos/testing-validation-tools.md new file mode 100644 index 000000000..8b4f36bd9 --- /dev/null +++ b/docs/how-tos/testing-validation-tools.md @@ -0,0 +1,175 @@ +# The shows the example of testing NeXus files with validation methods + +There are [different methods](validate-nexus-file.md), which can be used for file validation. + +- pynxtools (verify_nexus, read_nexus) +- nxvalidate +- punx + +Here some examples are shown for the respective methods, by using a pynxtools-ellips generated NeXus file. This generated file already contained some level of validation, as a generated and filled template for this NeXus application definition was used. + + +# 1. Example from pynxtools read_nexus function + +`read_nexus -f SiO2onSi.ellips.nxs > read_nexus_output_file.txt` +``` +NXellipsometry.nxdl.xml:/ENTRY/data_collection/data_software +NXprogram.nxdl.xml: +DEBUG: @url - IS NOT IN SCHEMA +#################################################### +NXellipsometry.nxdl.xml:/ENTRY/definition +NXoptical_spectroscopy.nxdl.xml:/ENTRY/definition +NXentry.nxdl.xml:/definition +DEBUG: @url - IS NOT IN SCHEMA +#################################################### +DEBUG: ===== GROUP (//entry/instrument/software_RC2 [NXellipsometry::/NXentry/NXinstrument/software_RC2]): +DEBUG: classpath: ['NXentry', 'NXinstrument'] +DEBUG: NOT IN SCHEMA +#################################################### +DEBUG: ===== FIELD (//entry/instrument/software_RC2/program): +DEBUG: value: b'CompleteEASE' +DEBUG: classpath: ['NXentry', 'NXinstrument'] +DEBUG: NOT IN SCHEMA +#################################################### +DEBUG: ===== ATTRS (//entry/instrument/software_RC2/program@url) +DEBUG: value: https://www.jawoollam.com/ellipsometry-software/completeease +DEBUG: classpath: ['NXentry', 'NXinstrument'] +DEBUG: NOT IN SCHEMA +#################################################### +DEBUG: ===== ATTRS (//entry/instrument/software_RC2/program@version) +DEBUG: value: 6.37 +DEBUG: classpath: ['NXentry', 'NXinstrument'] +DEBUG: NOT IN SCHEMA +#################################################### +``` +_Total 6 Errors_ +1. @url. Changing to @URL could fix this maybe. +2. Software_RC2 not detected as NXprogram. This is indeed not assigned. + +# 2. Example from pynxtools verify_nexus function + +` verify_nexus SiO2onSi.ellips.nxs` + +``` +WARNING: Field /entry/data_collection/Delta_50deg/@units written without documentation. +WARNING: Field /entry/data_collection/Delta_50deg_errors/@units written without documentation. +WARNING: Field /entry/data_collection/Delta_60deg/@units written without documentation. +WARNING: Field /entry/data_collection/Delta_60deg_errors/@units written without documentation. +WARNING: Field /entry/data_collection/Delta_70deg/@units written without documentation. +WARNING: Field /entry/data_collection/Delta_70deg_errors/@units written without documentation. +WARNING: Field /entry/data_collection/Psi_50deg/@units written without documentation. +WARNING: Field /entry/data_collection/Psi_50deg_errors/@units written without documentation. +WARNING: Field /entry/data_collection/Psi_60deg/@units written without documentation. +WARNING: Field /entry/data_collection/Psi_60deg_errors/@units written without documentation. +WARNING: Field /entry/data_collection/Psi_70deg/@units written without documentation. +WARNING: Field /entry/data_collection/Psi_70deg_errors/@units written without documentation. +WARNING: Missing attribute: "/ENTRY/DATA/@axes" +WARNING: Missing attribute: "/ENTRY/DATA/@signal" +Invalid: The entry `entry` in file `SiO2onSi.ellips.nxs` is NOT a valid file according to the `NXellipsometry` application definition. +``` +_Total 14 Errors_ +_Total 3 Errors - without documentation_ +1. Psi+Delta with Unit+Errors written without doc. +2. Data @axes + @signal. May not find NXdata? Attributes are present in .nxs file. +3. entry not valid in NXellips. + + +# 3. Example from nxvalidate + +``PATH_TO_NX_VALIDATE_EXE/nxvalidate -l PATH_TO_FAIRMAT_NEXUS_DEF/nexus_definitions/ PATH_TO_NEXUS_FILE/SiO2onSi.ellips.nxs` +``` +definition=NXellipsometry.nxdl.xml message="Data type mismatch, expected NX_BOOLEAN, got H5T_ENUM { H5T_STD_I8LE; "FALSE" 0; "TRUE" 1; }" nxdlPath=/NXentry/NXinstrument/NXlens_opt/data_correction sev=error dataPath=/entry/instrument/focussing_probes/data_correction dataFile=/home/ron/GitPynxtoolsValidation/nexus_files/SiO2onSi.ellips.nxs +definition=NXellipsometry.nxdl.xml message="Required group missing" nxdlPath=/NXentry/NXinstrument/NXbeam sev=error dataPath=/entry/instrument dataFile=/home/ron/GitPynxtoolsValidation/nexus_files/SiO2onSi.ellips.nxs +definition=NXellipsometry.nxdl.xml message="Required group missing" nxdlPath=/NXentry/NXinstrument/NXdetector sev=error dataPath=/entry/instrument dataFile=/home/ron/GitPynxtoolsValidation/nexus_files/SiO2onSi.ellips.nxs +definition=NXellipsometry.nxdl.xml message="Data type mismatch, expected NX_BOOLEAN, got H5T_ENUM { H5T_STD_I8LE; "FALSE" 0; "TRUE" 1; }" nxdlPath=/NXentry/NXsample/backside_roughness sev=error dataPath=/entry/sample/backside_roughness dataFile=/home/ron/GitPynxtoolsValidation/nexus_files/SiO2onSi.ellips.nxs +definition=NXellipsometry.nxdl.xml message="Required units attribute missing" nxdlPath=/NXentry/NXdata/measured_data sev=error dataPath=/entry/data_collection/measured_data dataFile=/home/ron/GitPynxtoolsValidation/nexus_files/SiO2onSi.ellips.nxs +definition=NXellipsometry.nxdl.xml message="Data type mismatch, expected NX_BOOLEAN, got H5T_STRING { STRSIZE H5T_VARIABLE; STRPAD H5T_STR_NULLTERM; CSET H5T_CSET_UTF8; CTYPE H5T_C_S1; }" nxdlPath=/NXentry/NXidentifier/is_persistent sev=error dataPath=/entry/experiment_identifier/is_persistent dataFile=/home/ron/GitPynxtoolsValidation/nexus_files/SiO2onSi.ellips.nxs +definition=NXellipsometry.nxdl.xml message="Required group missing" nxdlPath=/NXentry/NXdata sev=error dataPath=/entry dataFile=/home/ron/GitPynxtoolsValidation/nexus_files/SiO2onSi.ellips.nxs +definition=NXellipsometry.nxdl.xml message="Required units attribute missing" nxdlPath=/NXentry/NXprocess/depolarization sev=error dataPath=/entry/derived_parameters/depolarization dataFile=/home/ron/GitPynxtoolsValidation/nexus_files/SiO2onSi.ellips.nxs +9 errors and 85 warnings found when validating /home/ron/GitPynxtoolsValidation/nexus_files/SiO2onSi.ellips.nxs +``` +_Total 8 or 9 Errors_ +1. Datatype mismatch for Bools: "H5T_STRING" or "H5T_ENUM" instead of "NX_BOOLEAN". In NXlens_opt, backside_roughness and is_persistent. +2. NXbeam + NXdetector: Has a problem with " exists: [min, 1, max, infty]" +3. "/NXentry/NXdata/measured_data". Units are missing (unit should be NX_ANY). +4. It does not find the NXdata (in this file it is at /entry/data_collection/). +5. Depolarization is not assigned the unit NX_unitless. + +## nxvalidate Errors: + +1. "Data type mismatch, expected NX\_BOOLEAN" --> "NeXus interprets NX\_BOOLEAN differently than h5py. NeXus uses an integer of 1 byte for NX\_BOOLEAN. This is an int8 or uint8." --> [https://github.com/nexusformat/cnxvalidate/issues/34]() + +2. Required group missing for "NXbeam" and "NXdetector". Probem with NeXus requirement as given in the .yaml file by: "exists: [min, 1, max, infty]"? + +3. "Required units attribute missing" for entry/data\_collection/measured\_data --> ? unclear. Units are assigned in NeXus file. + +4. "Required group missing" for /entry ---> ? unclear. + + +## nxvalidate Warnings: + +I think warnings can be evoked by: (-t in front of the NeXus file): + +``` +~/FAIRmat/WorkshopNeXusValid02/nxvalidate/cnxvalidate/build$ ./nxvalidate -l /home/ron/FAIRmat/WorkshopNeXusValid02/nxvalidate/nexus_definitions/ -t SiO2onSi.ellips.nxs +``` + +Most of the warnings are not critical at all. Not sure if this is helpful at all: + +here are some examples of the "messages" of the warnings: + +1. "Optional group missing" + +2. "Optional field missing" + +3. "Optional attribute units missing" + +4. "Validating field" + +5. "Validating group" + +6. "Additional base class dataset name found" + +7. "Additional base class dataset address found" + +8. "Unknown dataset wavelength\_spectrum found" + +9. "Additional base class group notes of type NXnote found" + +10. "Additional base class group environment\_sample of type NXenvironment found" + + + + + + +# 4. Example from punx +`punx validate SiO2onSi.ellips.nxs` +Not possible, as only the NIAC NeXus definitoon can right now be used as reference. Unclear if the `punx install` functionality is working or still developed. + + +# Summary +| Error Message | origin | Error in .nxs file? | Error in validation tool? | +| ---------------- | ---| ------------------------ | ------------------------ | +| unit + error without doc | verify_nexus | ? | ? | +| no @signal @axes for NXdata | verify_nexus | no | yes | +| entry not valid in NXellips | verify_nexus | ? | ? | +| @url error | read_nexus | no | yes | +| Software_RC2 no NXprogram | read_nexus | yes | no | +| Bool Data types | nxvalidate | ? | ? | +| exists: [min, 1, max, infty] | nxvalidate | no | yes | +| Unit missing for measured_data| nxvalidate | yes | no | +| NXdata not present | nxvalidate | no | yes | +| No unit for depolarization | nxvalidate | yes | no | + + +### NOTE ### +Only the nxvalidate method seems to point out completely missing required concepts. + +I tested this with an empty NeXus file, in which only the "definition" was given (NXellipsometry and NXraman). + + + + + + diff --git a/docs/how-tos/validate-nexus-file.md b/docs/how-tos/validate-nexus-file.md new file mode 100644 index 000000000..2534bdb33 --- /dev/null +++ b/docs/how-tos/validate-nexus-file.md @@ -0,0 +1,773 @@ +# Validating NeXus files + +Note: This is a how-to guide for using different tools to validate NeXus files. If you want to learn more about how validation is done in `pynxtools`, please visit the [explanation page](../learn/nexus-validation.md). + +__The goal__ + +Use a tool to validate NeXus files to a given set of NeXus definitions: + + 1. [FAIRmat](https://fairmat-nfdi.github.io/nexus_definitions/index.html#) + + 2. [NIAC](https://manual.nexusformat.org/) + + +## Validation of a .nxs file + +The validity of NeXus files is fundamental to ensure FAIR data. Without specific requirements, it is not possible to understand the data. What type of experiment? What Laser Wavelength? Which voltage? What data is represented at all in the table? What is the unit of the value? Which ISO norm does this refer to? Where was this measured? Which year was this measured? + +The NeXus application definitions define the minimum set of terms that must be used in an instance of that class (i.e., the required terms that you must add to the file in order to be compliant with that application definition). Application definitions also may define terms that are optional in the NeXus data file. The requirements are set by the community via workshops or at conferences. To initiate or propose changes/additions, you can comment the FAIRMat NeXus proposal by going to the [NeXus definitions](https://fairmat-nfdi.github.io/nexus_definitions/index.html#), and using the hypothes.is tool (sign-up/log-in) to give us some feedback (Red boxes in the image. Expand this panel on the left by clicking on the arrow symbol). +![image.png](<./attachments/9d748230d54a8059-image.png>) + +Oftentimes, there will be errors in a generated NeXus file (be it by hand or automatically): Typos, missing required concepts, missing attributes, using the incorrect datatype or format (e.g., array instad of list, float instead of integer, etc.). Therefore, a validation is required, to ensure that the data you want to share is FAIR. + +The NeXus file is valid if it complies with the respective NeXus application definition. + +This validation is done by software. + + +### Validation software + +There are right now three tools, which can be used for validation of NeXus files. All are different and have individual advantages or disadvantages: + +1. pynxtools + +2. cnxvalidate + +3. punx + +Open software is usually shared on Github - There you find usually the most accurate information, as documentation sometimes lags behind. There you see a box with folders and files. Below is the content of the README.md file displayed. This usually shows instructions for installation and handling of the software. + +Here are the GitHub links for the three software packages: + +[pynxtools]() + +[cnxvalidate]() + +[punx]() + +In the following, each package and its capabilities is presented. + +### Operating systems + +Almost all PC users are used to Windows as operating system. + +A lot of software development is done on Linux as operating system. + +This is not a problem for big companies, but for smaller open software projects, which are often developed without funding, this is a problem. + +If you are used to Windows, consider setting up a Linux operating system to eliminate problems in the installation process and ensure compatibility. + + +## pynxtools + +pynxtools = Python Nexus Tools + +_[> learn more about validation in pynxtools <](../learn/nexus-validation.md)_ + +This is a python package which is developed by the FAIRmat consortium. + +As a python package, this can be used on Linux and Windows systems. + +The package can be installed via pip. Therefore you need to have installed: + +1. python + +2. pip + + + +For validation purposes, we will use the "read\_nexus" and "verify\_nexus" command line tools from `pynxtools`. + +## pynxtools - *verify_nexus* + +This tool is currently in development. It enables a command like: + +``` +verify_nexus C:\nexusvalidation\Raman.nxs +``` + + +The output warning looks like this: +``` +... +WARNING: Field /entry/instrument/beam_incident/wavelength/@units written without documentation. +... +``` + +### Installation of *verify_nexus* + +The `verify_nexus` function is currently under development (Aug 2024). Therefore, you have to install pynxtools from its feature branch until this function is published. +Do this to install pynxtools with verify\_nexus via; + +``` +pip install git+https://github.com/FAIRmat-NFDI/pynxtools@hdf-based-validation +``` + +Then, you should be able to call its help function: + +``` +verify_nexus --help +``` + +with this output: + +``` +Usage: verify_nexus [OPTIONS] FILE + + Verifies a nexus file + +Options: + --help Show this message and exit. +``` + + +### Using *verify_nexus* + +Open your terminal. Assuming there is a folder at: + +For Linux: + +``` +/home/USER/nexusvalidation +``` + +For Windows: + +``` +C:\nexusvalidation +``` + +Put into this folder your NeXus file, for example this [Raman.nxs file](https://zenodo.org/records/13373909/files/Raman.nxs?download=1). + +Use verify_nexus with the command: + +``` +verify_nexus C:\nexusvalidation\Raman.nxs +``` + +The respective output is: + +``` +WARNING: Field /entry/data/spectrum_data_x/@units written without documentation. +WARNING: Field /entry/data/spectrum_data_x_Raman/@units written without documentation. +WARNING: Field /entry/data/spectrum_data_y/@units written without documentation. +WARNING: Field /entry/instrument/beam_incident/wavelength/@units written without documentation. +WARNING: Field /entry/instrument/detector_DU970BV/number_of_cycles/@units written without documentation. +Invalid: The entry `entry` in file `Raman.nxs` is NOT a valid file according to the `NXraman` application definition. +``` + + +## pynxtools - *read_nexus* +While `verify_nexus´ is used as a straightforward tool for validating a NeXus file, `read_nexus` outputs a debug log for a given NeXus file by annotating the data and metadata entries with the schema definitions from the respective NeXus base classes and application definitions to which the file refers to. This can be helpful to extract documentation and understand the concept defined in the NeXus application definition. +The command used is: + +``` +read_nexus -f NXopt_minimal_example.nxs +``` + +The output looks like this, if the respective entry is found: + +``` +DEBUG: ===== FIELD (//entry/experiment_type): +DEBUG: value: b'transmission spectroscopy' +DEBUG: classpath: ['NXentry', 'NX_CHAR'] +DEBUG: classes: +NXoptical_spectroscopy.nxdl.xml:/ENTRY/experiment_type +DEBUG: <> +DEBUG: enumeration (NXoptical_spectroscopy.nxdl.xml:/ENTRY/experiment_type): +DEBUG: -> photoluminescence +DEBUG: -> transmission spectroscopy +DEBUG: -> reflection spectroscopy +DEBUG: -> other +DEBUG: documentation (NXoptical_spectroscopy.nxdl.xml:/ENTRY/experiment_type): +DEBUG: + Specify the type of the optical experiment. + + Chose other if none of these methods are suitable. You may specify + fundamental characteristics or properties in the experimental sub-type. + + For Raman spectroscopy or ellipsometry use the respective specializations + of NXoptical_spectroscopy. +``` + +or like this, if the respective entry is not found in the defintion: + +``` +DEBUG: ===== ATTRS (//entry/instrument/software_RC2/program@url) +DEBUG: value: https://www.jawoollam.com/ellipsometry-software/completeease +DEBUG: classpath: ['NXentry', 'NXinstrument'] +DEBUG: NOT IN SCHEMA +DEBUG: +``` + +The first example was for for "experiment\_type" entry in the "NXoptical\_spectroscopy" definition. + +The second example was for the "software\_TYPE" attribute @URL entry in the "NXoptical\_spectroscopy" definition. Here the problem was that "url" was used instead of "URL". + + + +### Installation of *read_nexus* + +This is installed with pip: + +``` +pip install pynxtools +``` + + +### Using *read_nexus* + +Open your terminal. Assuming there is a folder at: + +For Linux: + +``` +/home/USER/nexusvalidation +``` + +For Windows: + +``` +C:\nexusvalidation +``` + +Put into this folder your NeXus file, for example the [Raman.nxs file](https://zenodo.org/records/13373909/files/Raman.nxs?download=1). + +Then use: + +``` +read_nexus -f C:\nexusvalidation\Raman.nxs +``` + +shows the output like this: + +``` +===== FIELD (//entry/data/spectrum_data_y): +DEBUG: ===== FIELD (//entry/data/spectrum_data_y): +value: [ 288.5499878 289. 288.4500122 ... 1875. 1889.349976 ... +DEBUG: value: [ 288.5499878 289. 288.4500122 ... 1875. 1889.349976 ... +Dataset referenced as NXdata SIGNAL +DEBUG: Dataset referenced as NXdata SIGNAL +===== ATTRS (//entry/data/spectrum_data_y@long_name) +DEBUG: ===== ATTRS (//entry/data/spectrum_data_y@long_name) +value: Raman Intensity +DEBUG: value: Raman Intensity +Dataset referenced as NXdata SIGNAL +DEBUG: Dataset referenced as NXdata SIGNAL +===== ATTRS (//entry/data/spectrum_data_y@units) +DEBUG: ===== ATTRS (//entry/data/spectrum_data_y@units) +value: counts +DEBUG: value: counts + +DEBUG: +For Axis #0, 1 axes have been identified: [] +DEBUG: For Axis #0, 1 axes have been identified: [] +``` + +Search for fields which are not found in the NeXus definition by searching for the line: "DEBUG: NOT IN SCHEMA". Recheck the used NeXus definition to eliminate the problem. Be careful with upper and lower case notation and correct spelling. + +Keep in mind that the output provides quite some information. This is useful for software development, but may be a bit too much for validation purposes. + + + + + + + + + + + +## cnxvalidate + +This package is written in C. It is allows a command line evocation like: + +``` +nxvalidate -l appdefdir datafile +``` + +1. nxvalidate: calls the software function + +2. \-l appdefdir: points to the location of the NeXus definitions you want to use. This is a path to a folder called "defintions". + +3. datafile: This is the path to the .nxs file which should be checked. + +This output shows warnings like: + +``` +definition=NXoptical_spectroscopy.nxdl.xml message="Required attribute URL missing" nxdlPath=/NXentry/definition sev=error dataPath=/entry/definition dataFile=NXopt_minimal_example.nxs +``` + +and indicates the entry of the .nxs file, which is incorrect and what the respective problem is. It also points to the NeXus definition (.nxdl.xml file), in which this conflict was found. + +### Installation + +Note: You can find more information about installing nxvalidate [here](installation_notes_nxvalidate.md). Note that installation on Windows can be tricky because cmake can sometimes not find the libxml2 library. Though, if you solve this, this maybe work on windows). + +Therefore: Use linux. + +The software has to be built from source. This is eased significantly by using another software called: cmake. + +__Install cmake, github, hdf5 & xml2 library, etc:__ + +Open the terminal and install all parts required to install cnxvalidate via cmake: + +``` +sudo apt-get update +sudo apt-get install git +sudo apt-get install build-essential +sudo add-apt-repository universe +sudo apt-get install libhdf5-serial-dev +sudo apt-get -y install pkg-config +sudo apt upgrade -y +sudo apt-get -y install cmake +sudo apt-get install libxml2-dev +``` + +__Directory location__ + +Create a folder named "nexusvalidation" via terminal or file manager. + +The folder is located at `/home/USER/nexusvalidation` + +"USER" is your user name. (You can get your username by the terminal command: `echo $USER`) + +In the terminal, this is indicated by `~/nexusvalidation` ( \~ = /home/USER) + +Open the thermal and go into this directory by: + +``` +cd /home/USER/nexusvalidation +``` + +__Using GitHub__ + +Go to the [Github Repository of cnxvalidate:]() + +Click on the green "<> Code" button. + +Click on "HTTPS". + +Copy the https link. + +![image.png](<./attachments/e28ec15bb537c9b7-image.png>) + +Open the terminal and ensure you are in the `nexusvalidation` folder. + +Clone the github repository (= download the files of the software). + +``` +git clone https://github.com/nexusformat/cnxvalidate.git +``` + +now you have a new folder at `~/nexusvalidation/cnxvalidate` + +go into this folder via the command + +``` +cd cnxvalidate +``` + +now you are in the source tree. This should be exactly the same files, which you find on the github repository [github repository]() + +make a new directory called "build": + +``` +mkdir build +``` + +go into this directory + +``` +cd build +``` + +use cmake, to compile/build the software - this puts together all pieces of software - and especially external parts such as xml2 and hdf5 library. + +``` +cmake ../ +``` + +install cnxvalidate after it was sucessfully build + +``` +make +``` + +Now the above mentioned commands should be avaialble. The programm/executable is located at: + +``` +/home/USER/nexusvalidation/cnxvalidate/build/nxvalidate +``` + +### Using cnxvalidate + +Now you can start to validate your created [NeXus file](https://zenodo.org/records/13373909). But before the validation, we need to get a set of NeXus definitions, which we want to use as reference. This is done again by using git: + +__Getting NeXus definitions__ + +go to the folder nexusvalidation + +``` +cd /home/USER/nexusvalidation +``` + +Download a set of NeXus definitions. Choose only one: + +For FAIRmat NeXus definitions, go to [this link]() and copy the github "Code" line to clone the repository. Then: + +``` +git clone https://github.com/FAIRmat-NFDI/nexus_definitions.git +``` + +For the NIAC NeXus definitions, go to [this link]() and copy the github "Code" line to clone the repository. Then: + +``` +git clone https://github.com/nexusformat/definitions.git +``` + +Now you have a folder called "definitions" in the "nexusvalidation" folder. The path to this definitions folder is used as option for cnxvalidate, to tell the program which NeXus definitions shall be used. + +The respective path would be: + +``` +/home/USER/nexusvalidation/definitions +``` + +__Get your NeXus file__ + +put one of created NeXus file (or this [this file](https://zenodo.org/records/13373909/files/NXopt_minimal_example.nxs?download=1)) into the "nexusvalidation" folder (filemanager/explorer). + +The file should now be loacted at (assumed the file name is "NXopt_minimal_example.nxs") + +``` +/home/USER/nexusvalidation/NXopt_minimal_example.nxs +``` + +__Validating the NeXus file__ + +now you can use the cnxvalidate with the executable called "nxvalidate" to use the set of NeXus definitions called "appdefdir" to validate the NeXus file called "datafile". This is done from the terminal. + +``` +nxvalidate -l appdefdir datafile +``` + +All names are "paths" to the definition, application or file. Use the "full path", if you are not experienced, but relative paths work as well. + +For the provided example, the suitable command looks like: + +``` +/home/USER/nexusvalidation/cnxvalidate/build/nxvalidate -l /home/USER/nexusvalidation/definitions /home/USER/nexusvalidation/NXopt_minimal_example.nxs +``` + +The "-l" option tells the program, that it should look for the nexus definiton at the path after "-l". + +For the proved file above, the output should look like this: + +``` +USER@XXX:/home/USER/nexusvalidation/cnxvalidate/build/nxvalidate -l /home/USER/nexusvalidation/definitions /home/USER/nexusvalidation/NXopt_minimal_example.nxs +definition=NXoptical_spectroscopy.nxdl.xml message="Required attribute version missing" nxdlPath=/NXentry/definition sev=error dataPath=/entry/definition dataFile=NXopt_minimal_example.nxs +definition=NXoptical_spectroscopy.nxdl.xml message="Required attribute URL missing" nxdlPath=/NXentry/definition sev=error dataPath=/entry/definition dataFile=NXopt_minimal_example.nxs +definition=NXoptical_spectroscopy.nxdl.xml message="Required field missing" nxdlPath=/NXentry/experiment_type sev=error dataPath=/entry/experiment_type dataFile=NXopt_minimal_example.nxs +definition=NXoptical_spectroscopy.nxdl.xml message="Required group missing" nxdlPath=/NXentry/NXinstrument sev=error dataPath=/entry dataFile=NXopt_minimal_example.nxs +definition=NXoptical_spectroscopy.nxdl.xml message="Required group missing" nxdlPath=/NXentry/NXsample sev=error dataPath=/entry dataFile=NXopt_minimal_example.nxs +definition=NXoptical_spectroscopy.nxdl.xml message="Required group missing" nxdlPath=/NXentry/NXdata sev=error dataPath=/entry dataFile=NXopt_minimal_example.nxs +9 errors and 11 warnings found when validating NXopt_minimal_example.nxs +``` + +The errors tell you now which things are missing (message="Required group missing"), if there is a field missing (message="Required field missing"), or if an attribute is missing (message="Required attribute URL missing" - here for example the attribute named URL) + +Now adjust the file creation, and add the respective fields to make your NeXus file compliant with the NeXus definitions. This way, you can ensure that your data is FAIR, which is then ready for sharing and publication. + + + + + + + +## punx + +_punx - Python Utilities for NeXus HDF5 files_ + +This is python package, and can therefore be used on Linux and Windows systems. + +The package can be installed via pip. Therefore you need to have installed: + +1. python + +2. pip + +You can then evoke a command like this: + +``` +punx validate [-h] [--report REPORT] infile +``` + +"validate" tells the program that we want to validate a file + +"[-h]" tells the program to show the help message + +"[--report REPORT]" tells the program what findings should be reported. +This is done by replacing REPORT with ={COMMENT,ERROR,NOTE,OK,TODO,UNUSED,WARN} + +[Official docs]() + +### Installation + +Open the terminal and install punx via pip: + +``` +pip install punx +``` + +This software is based on other powerful software packages or libraries, therefore as well other packages have to be installed: + +``` +pip install h5py +pip install lxml +pip install numpy +pip install PyQt5 +pip install requests +pip install pyRestTable +``` + +Then you should be able to test the package by: + +``` +punx demo +``` + +The output should look like this: + +``` +C:\>punx demo + +!!! WARNING: this program is not ready for distribution. + + +console> punx validate C:\Users\USER\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\punx\data\writer_1_3.hdf5 +data file: C:\Users\USER\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\punx\data\writer_1_3.hdf5 +NeXus definitions: main, dated 2024-01-02 03:04:05, sha=xxxx21fxcef02xfbaa6x04e182e3d67dace7ef1b + +findings +============================ ======== ==================================== ========================================================== +address status test comments +============================ ======== ==================================== ========================================================== +/ TODO NeXus base class NXroot: more validations needed +/ OK known NXDL NXroot: recognized NXDL specification +/ OK NeXus base class NXroot: known NeXus base class +/ OK NeXus default plot found by v3: /Scan/data/counts +/ OPTIONAL NXDL group in data file not found: in //entry +/Scan TODO NeXus base class NXentry: more validations needed +/Scan OK group in base class not defined: NXroot/Scan +/Scan OK known NXDL NXentry: recognized NXDL specification +/Scan OK NeXus base class NXentry: known NeXus base class +/Scan OK NXDL group in data file found: in /Scan/data +/Scan NOTE validItemName relaxed pattern: [a-zA-Z0-9_]([a-zA-Z0-9_.]*[a-zA-Z0-9_])? +/Scan OPTIONAL NXDL field in data file not found: /Scan/collection_description +/Scan OPTIONAL NXDL field in data file not found: /Scan/collection_identifier +/Scan OPTIONAL NXDL field in data file not found: /Scan/collection_time +/Scan OPTIONAL NXDL field in data file not found: /Scan/definition +/Scan OPTIONAL NXDL field in data file not found: /Scan/definition_local +... +... +... +/Scan/data@signal OK known attribute known: NXdata@signal +/Scan/data@signal OK value of @signal found: /Scan/data/counts +/Scan/data@signal OK NeXus default plot v3, NXdata@signal correct default plot setup in /NXentry/NXdata +/Scan/data@two_theta_indices TODO attribute value implement +/Scan/data@two_theta_indices OK validItemName strict pattern: [a-z_][a-z0-9_]* +/Scan/data@two_theta_indices OK known attribute unknown: NXdata@two_theta_indices +/Scan/data/counts OK validItemName strict pattern: [a-z_][a-z0-9_]* +/Scan/data/counts OK field in base class not defined: NXdata/counts +/Scan/data/counts@units TODO attribute value implement +/Scan/data/counts@units OK validItemName strict pattern: [a-z_][a-z0-9_]* +/Scan/data/two_theta OK validItemName strict pattern: [a-z_][a-z0-9_]* +/Scan/data/two_theta OK field in base class not defined: NXdata/two_theta +/Scan/data/two_theta@units TODO attribute value implement +/Scan/data/two_theta@units OK validItemName strict pattern: [a-z_][a-z0-9_]* +============================ ======== ==================================== ========================================================== + + +summary statistics +======== ===== =========================================================== ========= +status count description (value) +======== ===== =========================================================== ========= +OK 35 meets NeXus specification 100 +NOTE 1 does not meet NeXus specification, but acceptable 75 +WARN 0 does not meet NeXus specification, not generally acceptable 25 +ERROR 0 violates NeXus specification -10000000 +TODO 7 validation not implemented yet 0 +UNUSED 0 optional NeXus item not used in data file 0 +COMMENT 0 comment from the punx source code 0 +OPTIONAL 40 allowed by NeXus specification, not identified 99 + -- +TOTAL 83 +======== ===== =========================================================== ========= + +=99.144737 of 76 items reviewed +NeXus definitions version: main + +console> punx tree C:\Users\rh83hixu\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\punx\data\writer_1_3.hdf5 +C:\Users\rh83hixu\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\punx\data\writer_1_3.hdf5 : NeXus data file + Scan:NXentry + @NX_class = "NXentry" + data:NXdata + @NX_class = "NXdata" + @axes = "two_theta" + @signal = "counts" + @two_theta_indices = [0] + counts:NX_INT32[31] = [1037, 1318, 1704, '...', 1321] + @units = "counts" + two_theta:NX_FLOAT64[31] = [17.92608, 17.92591, 17.92575, '...', 17.92108] + @units = "degrees" +``` + +Then you should be able to use this package. + +Official docs for [punx installation]() + +### Using punx + +Open your terminal. Assuming there is a folder at: + +For Linux: + +``` +/home/USER/nexusvalidation +``` + +For Windows: + +``` +C:\nexusvalidation +``` + +Put a NeXus file into this folder. For example, the file: [SiO2onSi.ellips.nxs](https://zenodo.org/records/13373909/files/SiO2onSi.ellips.nxs?download=1). + +then the command is (for Windows): + +``` +punx validate C:\nexusvalidation\SiO2onSi.ellips.nxs +``` + +For Linux: + +``` +punx validate C:\nexusvalidation\SiO2onSi.ellips.nxs +``` + +The output tables "findings" and "summary statistics" can be used to find error present in the NeXus file. + +__Which NeXus definition?__ + +The program selects the NeXus definitions (set of nxdl.xml files) by itself. It can in principle also be modified with different repositories. The functionality to add a new repository is right now not possible (Aug 2024). + +Therefore, only the NIAC repository as NeXus definitions is functional. + +You may update the repository for the lastest version via: + +``` +punx install +``` + +The NeXus respective definitions are found here: + +[NIAC NeXus definitions]() + +Search on the right side under "quick search" for "NXopt": + +[NXopt NeXus definition]() + +This python code creates the respective python file with all required fields: + +[NXopt_minimal_example_NIAC_NeXus_Def.nxs](https://zenodo.org/records/13373909/files/NXopt_minimal_example_NIAC_NeXus_Def.nxs?download=1) + +Here is the python code: + +[h5py_nexus_file_creation_NIAC_NeXus_Def.py](https://zenodo.org/records/13373909/files/h5py_nexus_file_creation_NIAC_NeXus_Def.py?download=1) + +The command: + +``` +punx validate --report ERROR C:\nexusvalidation\NXopt_minimal_example_NIAC_NeXus_Def.nxs +``` + +then gives this output: + +``` +findings +======= ====== ========== ====================================== +address status test comments +======= ====== ========== ====================================== +/entry ERROR known NXDL NXopt: unrecognized NXDL specification +======= ====== ========== ====================================== + + +summary statistics +======== ===== =========================================================== ========= +status count description (value) +======== ===== =========================================================== ========= +OK 148 meets NeXus specification 100 +NOTE 0 does not meet NeXus specification, but acceptable 75 +WARN 0 does not meet NeXus specification, not generally acceptable 25 +ERROR 1 violates NeXus specification -10000000 +TODO 16 validation not implemented yet 0 +UNUSED 0 optional NeXus item not used in data file 0 +COMMENT 0 comment from the punx source code 0 +OPTIONAL 213 allowed by NeXus specification, not identified 99 + -- +TOTAL 378 +======== ===== =========================================================== ========= +``` + +The last error message: + +``` +======= ====== ========== ====================================== +/entry ERROR known NXDL NXopt: unrecognized NXDL specification +======= ====== ========== ====================================== +``` + +can be ignored and is a bug right now. If this is the only Error message, then your NeXus file is compliant with the NeXus definitions and you can share and publish your data. + +### Further notes + +1. Punx only uses the NeXus definiton from the NIAC [NeXus definiton from the NIAC](). The use of the [FAIRmat NeXus definition]() is not possible right now. + +2. [Other punx commands are availble]() + +3. [More details for installation]() + +4. [Github project]() + + + + +## Summary + +This tutorial showed: + +1. 3 different tools for NeXus file validation + +2. How to install these tools + +3. How to use them via Examples + +### Recommended workflow + +As pynxtools verify_nexus method is right now in development, [not all situations are covered right now](testing-validation-tools.md). Therefore, the most reliable method right now is a combination of _Human Manual Validation_ + _Software solutions_. + +### Pynxtools Parsers + +For a specifically structured set of data, a parser can be written, which uses the meta data and a pre-structured meta data file, to create a NeXus file. Tough, the parser depends on: Experimental Technique and Setup and has therefore to be written individually. Take a look [here](../reference/plugins.md). + + +### Feedback and contact + +1. Best way is to contact the software development team directly via a [Github Issue](https://github.com/FAIRmat-NFDI/pynxtools/issues/new). + +2. ron.hildebrandt(at)physik.hu-berlin.de + + + diff --git a/docs/index.md b/docs/index.md index 4b436eedd..5a5becf9e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -35,12 +35,18 @@ A series of tutorials giving you an overview on how to store or convert your dat How-to guides provide step-by-step instructions for a wide range of tasks. -- [Writing an application definition](how-tos/writing-an-appdef.md) -- [Storing data in multiple application definitions](how-tos/using-multiple-appdefs.md) + - [Build your own pynxtools plugin](how-tos/build-a-plugin.md) - [Implement a reader based on the MultiFormatReader](how-tos/use-multi-format-reader.md) +- [Validation of NeXus files](how-tos/validate-nexus-file.md) +- [Creation of NeXus files in python via hard-coding](how-tos/create-nexus-files-by-python.md) +- [Using pynxtools test framework for plugins](how-tos/using-pynxtools-test-framework.md) + +__The following How-To guides are still under development:__ + +- [Writing an application definition](how-tos/writing-an-appdef.md) +- [Storing data in multiple application definitions](how-tos/using-multiple-appdefs.md) - [Representing experimental geometries](how-tos/transformations.md) -- [Using pynxtools test framework](how-tos/using-pynxtools-test-framework.md) @@ -53,6 +59,7 @@ How-to guides provide step-by-step instructions for a wide range of tasks. - [An introduction to NeXus](learn/nexus-primer.md) - [Rules for storing data in NeXus](learn/nexus-rules.md) - [The concept of multiple application definitions](learn/multiple-appdefs.md) +- [The MultiFormatReader as a reader superclass](learn/multi-format-reader.md) #### pynxtools diff --git a/docs/learn/nexus-validation.md b/docs/learn/nexus-validation.md index 02a4b91a5..67a036e2e 100644 --- a/docs/learn/nexus-validation.md +++ b/docs/learn/nexus-validation.md @@ -1,5 +1,7 @@ # NeXus validation -!!! info "Work in progress" +!!! danger "Work in progress" + +!!! info "This page is intended to give more information about the validation tools that are part of `pynxtools`. Please also have a look at our comprehensive [how-to guide](../learn/nexus-validation.md) on NeXus validation." One of the main advantages of using pynxtools is that it comes with its own validation tools. That is, it can be used to validate that a given NeXus/HDF5 file is compliant with a NeXus application definition. @@ -38,7 +40,9 @@ MSYS_NO_PATHCONV=1 read_nexus -c /NXarpes/ENTRY/INSTRUMENT/analyser This workaround was tested with Windows 11, but should very likely also work with Windows 10 and lower. ## Other approaches (not part of pynxtools) -Aside from the tools we developed within FAIRmat, the [official NeXus website](https://manual.nexusformat.org/validation.htm) listed two more programs for the validation of NeXus files: +Aside from the tools we developed within FAIRmat, the [official NeXus website](https://manual.nexusformat.org/validation.htm) lists two more programs for the validation of NeXus files: + +1. [cnxvalidate]() +2. [punx]() -1. nxvalidate -2. punx \ No newline at end of file +We will not discuss the details of these two programs here, but you can find some information about the in the how-to guide linked above. \ No newline at end of file diff --git a/mkdocs.yaml b/mkdocs.yaml index 11f03ade5..9dcb7d970 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -13,10 +13,12 @@ nav: - How-tos: - how-tos/writing-an-appdef.md - how-tos/using-multiple-appdefs.md + - how-tos/transformations.md - how-tos/build-a-plugin.md - how-tos/use-multi-format-reader.md - - how-tos/transformations.md - how-tos/using-pynxtools-test-framework.md + - how-tos/create-nexus-files-by-python.md + - how-tos/validate-nexus-file.md - Learn: - NeXus: - learn/nexus-primer.md diff --git a/src/pynxtools/dataconverter/README.md b/src/pynxtools/dataconverter/README.md index c452fb472..e7239c2af 100644 --- a/src/pynxtools/dataconverter/README.md +++ b/src/pynxtools/dataconverter/README.md @@ -14,4 +14,4 @@ In order to understand the dataconverter, these pages might be particularly help - [Tutorial: Converting research data to NeXus](https://fairmat-nfdi.github.io/pynxtools/tutorial/converting-data-to-nexus.html) - [Built-in readers of pynxtools](https://fairmat-nfdi.github.io/pynxtools/reference/built-in-readers.html) - [Existing reader plugins for different experimental techniques](https://fairmat-nfdi.github.io/pynxtools/reference/plugins.html) -- [How-to guide for writing your own reader/plugin](https://fairmat-nfdi.github.io/pynxtools/how-tos/build-a-plugin.html) \ No newline at end of file +- [How-to guide for writing your own reader/plugin](https://fairmat-nfdi.github.io/pynxtools/how-tos/build-a-plugin.html)