Merge branch 'master' into verification

.github/workflows/build_docs.yml

@@ -0,0 +1,27 @@
+name: build_docs
+on:
+  push:
+    branches: [master]
+permissions:
+  contents: write
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Configure Git Credentials
+        run: |
+          git config user.name github-actions[bot]
+          git config user.email 41898282+github-actions[bot]@users.noreply.github.com
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+      - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+      - uses: actions/cache@v3
+        with:
+          key: mkdocs-material-${{ env.cache_id }}
+          path: .cache
+          restore-keys: |
+            mkdocs-material-
+      - run: pip install -r mkdocs-requirements.txt
+      - run: mkdocs gh-deploy --force

.github/workflows/check_nexus_version.yml

@@ -0,0 +1,24 @@
+name: nexus version check
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+jobs:
+  linting:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Generate nexus git version
+        run: |
+          git submodule sync --recursive
+          git submodule update --init --recursive --jobs=4
+          git submodule foreach --recursive 'git fetch --tags'
+          cd pynxtools/definitions
+          git describe --dirty --tags --long --match "*[0-9]*" > ../nexus-ref.txt
+          cd ../
+      - name: Diff
+        run: |
+          diff -w pynxtools/nexus-version.txt pynxtools/nexus-ref.txt

.gitignore

@@ -203,6 +203,8 @@ cython_debug/
 !dev-requirements.txt
 !pynxtools/dataconverter/units/default_en.txt
 !pynxtools/dataconverter/units/constants_en.txt
+!mkdocs-requirements.txt
+!pynxtools/nexus-version.txt
 build/
 nexusparser.egg-info/PKG-INFO
 .python-version

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "pynxtools/definitions"]
 	path = pynxtools/definitions
-	url = https://github.com/FAIRmat-NFDI/nexus_definitions.git
+	url = https://github.com/FAIRmat-NFDI/nexus_definitions.git

.vscode/settings.json

@@ -6,8 +6,8 @@
   "[python]": {
     "editor.formatOnSave": true,
     "editor.codeActionsOnSave": {
-      "source.fixAll": true,
-      "source.organizeImports": true
+      "source.fixAll": "explicit",
+      "source.organizeImports": "explicit"
     },
     "editor.defaultFormatter": "charliermarsh.ruff"
   },

README.md

@@ -43,6 +43,10 @@ data into the NeXus standard and visualising the files content.
 
 - [**dataconverter**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/pynxtools/dataconverter/README.md): Creates compliant instances of NeXus/HDF5 files to [NeXus schemas](https://nexusformat.org).
 - [**read_nexus**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/pynxtools/nexus/README.md): Outputs a debug log for a given NeXus file.
+- [**generate_eln**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/pynxtools/eln_mapper/README.md): Outputs ELN files that can be used to add metadata to the dataconverter routine.
+
+# Documentation
+Documentation for the different tools can be found [here](https://fairmat-nfdi.github.io/pynxtools/).
 
 # Contributing
 

docs/how-tos/transformations.md

@@ -0,0 +1,5 @@
+# Storing experimental geometries
+
+!!! danger "Work in progress"
+
+    This part of the documentation is still being written and it might be confusing or incomplete.

docs/how-tos/using-multiple-appdefs.md

@@ -0,0 +1,5 @@
+# Storing data following multiple appdefs
+
+!!! danger "Work in progress"
+
+    This part of the documentation is still being written and it might be confusing or incomplete.

docs/how-tos/writing-an-appdef.md

@@ -0,0 +1,26 @@
+# Writing a Simple Application Definition
+
+!!! danger "Work in progress"
+
+    This part of the documentation is still being written and it might be confusing or incomplete.
+
+Concept of this How-to:
+
+Create an example file NXdouble_slit
+
+NXslit_experiments --> NXdouble_slit
+NXslit_experiments --> NXsingle_slit
+
+They should learn the basic principles of how nexus works, the different path notations
+- Principles of nexus
+    - concepts
+    - appdefs
+    - base classes
+- Inheritance of application definitions and base classes
+- Connection of concept paths and instance paths
+- Description of appdef/base class notation (upper and lower case)
+- Basic tools for creation (pynxtools) and verification (pynxtools?) of nexus files
+
+Additional information (i.e., not in this tutorial but linked to this):
+- Creating a reader in pynxtools
+- Reading/writing nexus data in nomad

docs/index.md

@@ -0,0 +1,62 @@
+---
+hide: toc
+---
+
+# FAIRmat NeXus documentation
+
+<!-- A single sentence that says what the product is, succinctly and memorably -->
+
+<!-- A paragraph of one to three short sentences, that describe what the product does. -->
+
+<!-- A third paragraph of similar length, this time explaining what need the product meets -->
+
+<!-- Finally, a paragraph that describes whom the product is useful for. -->
+
+<div markdown="block" class="home-grid">
+<div markdown="block">
+
+### Tutorial
+
+A series of tutorials giving you an overview on how to store or convert your data to NeXus compliant files.
+
+- [Converting your data to NeXus](tutorial/converting-data-to-nexus.md)
+- [Uploading NeXus data to NOMAD](tutorial/nexus-to-nomad.md)
+
+</div>
+<div markdown="block">
+
+### How-to guides
+
+How-to guides provide step-by-step instructions for a wide range of tasks:
+
+- [Writing an application definition](how-tos/writing-an-appdef.md)
+- [Storing data in multiple application definitions](how-tos/using-multiple-appdefs.md)
+- [Representing experimental geometries](how-tos/transformations.md)
+
+</div>
+
+<div markdown="block">
+
+### Learn
+
+An introduction to NeXus and its design principles.
+
+- [An introduction to NeXus](learn/nexus-primer.md)
+- [The concept of multiple application definitions](learn/multiple-appdefs.md)
+
+
+
+</div>
+<div markdown="block">
+
+### Reference
+
+[Here](reference/definitions.md), you find the detailed list of application definitions and base classes and their respective fields.
+
+Or go directly to the [official NIAC](https://manual.nexusformat.org/classes/index.html)
+ or [latest FAIRmat](https://fairmat-nfdi.github.io/nexus_definitions/) definitions.
+
+</div>
+</div>
+
+<h2>Project and community</h2>

docs/learn/multiple-appdefs.md

@@ -0,0 +1,27 @@
+# Multiple Application Definitions in NeXus
+
+!!! danger "Work in progress"
+
+    This part of the documentation is still being written and it might be confusing or incomplete.
+
+This tutorial showcases how to employ multiple application definitions in NeXus for creating a file that conforms to various definitions simultaneously.
+
+!!! info "Prerequisites"
+
+    Familiarity with the basics of NeXus and its application definitions is required. For an introduction to NeXus, please refer to the [basic documentation](nexus-primer.md).
+
+In a laboratory setting, the data we collect can vary significantly depending on the experiment's specific setup. Consider, for instance, an experiment characterized using the `NXexperiment` application definition. Suppose we want to enhance this experiment by incorporating energy resolution details. A straightforward approach might involve creating a specialized sub-application definition, like `NXexperiment_energy_resolved`, to include metadata about the experiment's energy resolution.
+
+While this method is effective for initial expansions of the metadata, it becomes cumbersome when trying to merge multiple enhancements into a single application definition. Imagine we wish to integrate additional elements that provide time resolution data for our experiment. 
+We need to create three sub application definitions to reflect all combinations: `NXexperiment_time_resolved`, `NXexperiment_energy_resolved` and `NXexperiment_energy_time_resolved`.
+For three experimental facets we already need 7 sub application definitions. An additional problem is that we have to repeat the whole procedure for all experiments we like to add the specific traits to. So if we have three different parent application definitions we already need to create 9 sub application definitions just to add energy and time resolution.
+
+The solution for this problem is to specify multiple application definitions while writing a NeXus file to follow different traits of the experiment. This allows us to simply create `NXtime_resolved` and `NXenergy_resolved` and combine it with any experiment we want to use it with.
+This comes, however, with a few drawbacks. One of them is that it is currently not possible to write a file which wants to use two different application definitions which have conflicting fields. While this is generally possible in the framework of NeXus, since every application definition creates their own namespace, this is not supported when the paths are reduced to entry path notation.
+
+ToDo:
+- Make an example of NXexperiment, NXtime_resolved and NXenergy_resolved and show how it is combined into the instance path.
+- Also show this for a conflict. Compare concept path (no problem) to instance path (colliding).
+- Write a part how it is described in the file that it follows two appdefs `/entry/definitions` as array containing both appdefs.
+- Explain that this is no problem with the expanded concept path notation but fails when we only use the instance path.
+- Explain the reader concept: One reader for one appdef, then pynxtools will figure out how to combine them (this is domain knowledge for the FAIRmat reader -> will be different when a read/write tool is written somewhere else).

docs/learn/nexus-primer.md

@@ -0,0 +1,38 @@
+# A primer on NeXus
+
+!!! danger "Work in progress"
+
+    This part of the documentation is still being written and it might be confusing or incomplete.
+
+NeXus is is a description of a common data exchange format initially developed for neutron, X-ray, and muon experiments. Within FAIRmat we extensively extended the format to cover a range of experiments with major support for APM, ARPES, XPS, and optical spectroscpy but we also give advice and guidance for developing standards for other formats as well.
+
+!!! info "NeXus as a tool for FAIR data"
+
+    NeXus is supported be the research data management platform NOMAD.
+    Experimental data following an NeXus application definition can easily be uploaded and is recognized by NOMAD's search system.
+    If you want to learn more about uploading NeXus data to NOMAD, please refer to the [NeXus to nomad](../tutorial/nexus-to-nomad.md) tutorial
+    of this documentation.
+    Accordingly, if you want to build data according to the FAIR principles, you can think of NeXus fulfilling the interoperability and
+    reproducibility part and a research data management platform like NOMAD the findable and accessible part.
+
+## What is NeXus?
+
+Sometimes, NeXus is seen as writing data to some form of file in hdf5 format.
+While this is partly true, NeXus is independent of the actual storage format but is typically written into an hdf5 file.
+
+But what is NeXus then? It is the conceptual layer above the file structure.
+It is a contract on which data has to be present and how to name them in a given dataset.
+Hence, using NeXus participates in making data FAIR.
+It especially covers the interoperability and reproducibility part of research data.
+
+!!! info "NeXus path notations"
+
+    There are several methods for referencing concepts or data paths within NeXus:
+
+    - **Concept Path Notation:** This notation describes the hierarchical structure of NeXus concepts using class names. For example, `NXexperiment:/NXentry/NXinstrument/NXdetector` indicates the creation of a new NXdetector class within the NXexperiment concept. This path typically forms automatically when an application definition extends a base class's fields.
+
+    - **Instance Path Notation:** It represents the actual location of a field or group in a NeXus data instance (e.g., an HDF5 file). An example is `my_file.nxs:/entry/instrument/detector`.
+
+    - **Combined Notation:** This combines concept and instance paths. For example, `NXexperiment:/NXentry[my_file.nxs:entry]/NXinstrument[instrument]/NXdetector[detector]`. Here, concept paths are outside and instance paths within square brackets. The leftmost entries may include the NeXus class or file reference.
+
+    - **Appdef Notation:** This format is used in application definitions, where uppercase indicates a selectable name and lowercase a fixed name. Examples include `NXexperiment:ENTRY[my_experiment.nxs:entry]/INSTRUMENT[instrument]/DETECTOR[detector]` and `NXexperiment:ENTRY[my_experiment.nxs:entry]/my_INSTRUMENT[my_instrument]/DETECTOR[detector]`.

docs/macros.py

@@ -0,0 +1,17 @@
+"""
+MKdocs macros for the documentation
+"""
+
+
+def define_env(env):
+    """
+    This is the hook for defining variables, macros and filters
+
+    - variables: the dictionary that contains the environment variables
+    - macro: a decorator function, to declare a macro.
+    - filter: a function with one of more arguments,
+        used to perform a transformation
+    """
+
+    # add to the dictionary of variables available to markdown pages:
+    env.variables["version"] = "2023.10"  # Figure out from setuptools-scm eventually

docs/reference/definitions.md

@@ -0,0 +1,12 @@
+# NeXus definitions
+
+We link two references here.
+The first links to the official definitions by the [NIAC](http://www.nexusformat.org) and the second one links to latest FAIRmat definitions.
+
+- [Official NIAC definitions](https://manual.nexusformat.org/classes/index.html)
+- [Latest FAIRmat definitions](https://fairmat-nfdi.github.io/nexus_definitions/)
+
+The FAIRmat definitions are regularly contributed to NIAC (around every 6 months) but generally reflect
+a state which is still under development and may contain new or improved application definitions or base classes.
+Consider it as the public review stage of these application definitions.
+However, there might be some parts which are still under discussion and will be subject to change.

docs/stylesheets/extra.css

@@ -0,0 +1,69 @@
+
+.md-header__button.md-logo :where(img,svg) {
+    width: 100%;
+    height: 30px;
+}
+
+.md-header, .md-header__inner {
+    background-color: #fff;
+    color: #2A4CDF;
+}
+
+.md-header[data-md-state=shadow] {
+    box-shadow: 0px 2px 4px -1px rgb(0 0 0 / 20%), 0px 4px 5px 0px rgb(0 0 0 / 14%), 0px 1px 10px 0px rgb(0 0 0 / 12%);
+    transition: box-shadow 200ms linear;
+}
+
+.md-header__inner {
+    height: 80px;
+}
+
+.md-header__topic {
+    font-size: 24px;
+}
+
+.md-footer {
+    background-color: #2A4CDF;
+}
+
+.md-search__form:hover {
+    background-color: rgba(0,0,0,.13);
+}
+
+.md-typeset h1 {
+    color: black;
+    font-weight: 700;
+}
+
+.youtube {
+    position: relative;
+    width: 100%;
+    height: 0;
+    padding-bottom: 56.25%;
+}
+
+.youtube iframe {
+    position: absolute;
+    top: 0;
+    left: 0;
+    width: 100%;
+    height: 100%;
+}
+
+.home-grid {
+    display: grid;
+    grid-template-columns: 1fr 1fr;
+    grid-column-gap: 24px;
+    row-gap: 24px;
+}
+
+.home-grid div {
+    border-radius: 4px;
+    padding: 24px;
+    background-color: #f3e9d9;
+}
+
+.home-grid h3 {
+    margin-top: 0;
+    font-weight: 700;
+}