Yaml2nxdl (#3)

-pylinting
-tools: yaml2nxdl; nxdl2yaml
-tools: dataconverter + readers for (MPES;Ellips;APM;EM)
-reference NeXus definitions with new classes 

* Skeleton setup to start development

* Added documentation and diagram to show convert routine.

* Created a reader notebook for metadata

* Made changes to the reader function

A basic structure for the reader function.

* Added the reader function to a class.

* Skeleton setup to start development

* Added documentation and diagram to show convert routine.

* Added writer functionality

* Updated docs

* first commit, create yaml2nxdl folder

* upload of yaml2nxdl files from Markus K.

* Update README.md

* removal of lxml, extension of README.md, bug fixes

* deletion of old files

* bug fixes, test folder generated

* test data added and testing case started

* fixed xml header

* nested symbols tested and fixed, xml stylesheet added

* bug fixes, testing cases for links ans symbols

* Update yaml2nxdl.py

documentation, transfer of issues detected by @mkuehbach

* Update yaml2nxdl.py

minor

* Update yaml2nxdl_recursive_build.py

added fixes from @mkuehbach

* Update README.md

added documentation

* Update README.md

typos and restructuring based on most relevant points first

* sintax error fixed in yaml2nxdl.py, exception added for xml_declaration not supported in python<=3.7

* changed xml output filename, created get_nx_classes function in read_nexus

* updated the reader function.

* yaml2nxdl_utils update

* Quick fix for NX_UNITLESS

* changed format to .py

* codestyle modification

* codestyle mod #2

* codestyle mod #3

* codestyle mod 4

* codestyle mod 5

* definitions from fork

* codestyle mod 6

* nexus definition submodule

* Removing install errors and updated Readme.md to include recursive git clone

* Added @Units detail to Readme.md

* Added tests

* new documentation handling

* Quick fix to accept nxdl names with underscores

* pylint warning fixes

* Uploaded dummy preprocessed data file

Data file with normal structure of the .h5 output of our preprocessing routine, but only 2D data to perfectly match the application definition.

* A working structure for the reader.

A few custom changes in the repo in this branch have been made for it to work in my PC (windows). Clone this version to get proper functionality.

* fixed path in read_nexus to access the nxdl.

* import fixes

* solved some pylint warnings

* Throws error when attributes couldn't be found and lists the path

* pylint warnings

* pylinting true in setting.json

* added a test for Nexus classes, attributes, units in /tests/test_nexus

* Changed path concatenation to use the os module

* Renamed read_nexus.py to nexus.py

* Fix writing NX_CLASS attribute to NX_class

* migrated read_nexus to nexus in yaml2nxdl

* fixed bug for \@  attribute identifiers

* pylint compliant modification

* fixed bug with attribute documentation, added --verbose option

* yaml2ndxl tests updated, doc strings style changed

* updated test files

* minor changes to doc strings

* A save in between

* Removed lxml and improved func to get nxdl attrs

* yaml2nxdl base classes parsing

* Added data and type validation

* adding nxdl2yaml tool, updated README.md

* changed argument parsing in nxdl2yaml - updated README

* Allowed lists as values for data_dict, type validation for lists, automated readers pytest

* added tests for nxdl2yaml

* Added tests for helper functions and quick fix for data validation

* partial linting of nexusparser tools

* pylinting

* Fixed issue with recommended check and added NX_UINT types

* Allowed optional values from readers to be sent back as None

* Fixed double import on test file and various fixes

* Added test cases for optional and recommended not given

* Added support for local timezones in iso8601 with tests

* yaml2nxdl keyword_type and verbose fix

* Added reader functionality for a test file

* test_parser and yaml2nxdl NX_COMPLEX fixes

* running version of nexus converter
added test files to tests folder
x

* further pylinting - overall code rate 9.69

* Allowed complete optional groups to be left out from filled in dict.

* merged nexus.py of yaml2nxdl branch, and fixed bug in get_nxdl_doc

* add xarray as requirement
bugfixes in test data and config_file. Output of current version can be checked against nexus.py tool

* add .gitignore and clean up repository

* finish cleanup

* Fixed W503

* Quick fix with Andrea

* Fixed a few paths in nexus.py for windows

* Em and apm reader (#7)

* Sprint5 result, apm and em_nion readers

* Provided NXDL xml files for them to be moved into nexusparser/dependencies/applications directory

* Fixed specific versions for odfpy and ase

* Fixed typo while fixing specific versions for odfpy and ase

* Deleted NXDL schema, will be send via e-mail

* Removed incorrect exceptions from reader.py

* Working mpes reader

* README update

* pytest settings settings.json

* Working ellipsometry reader

* test_parser fixes

* Fixed test_parser, test_nexus

* Requirements.txt

* Fixed tests after merge

* Skipped ApmReader and EmNionReader from pytest

* Added odfpy and ase as requirements for ApmReader/EmNionReader

* reference the new fairmat branch

* Fixed pytest-timeout requirement version

* Changed Ellipsometry data to one without embargo

* Updated definitions submodule to include NXcollectioncolumn.nxdl

* jupyter notebook for ellipsometry

* Fixes on apm and em_nion readers for issue91_demo

* adjust paths to new file location

* load own nxs file into h5web

* Apm and Em jupyter notebooks, paths for EM do not work for now, EM.ipynb NEEDS TO BE EXECUTED from within / !

Co-authored-by: Sherjeel Shabih <[email protected]>
Co-authored-by: Arora0 <[email protected]>
Co-authored-by: Aman Abdullayev <[email protected]>
Co-authored-by: Markus Kühbach <[email protected]>
Co-authored-by: sanbrock <[email protected]>
Co-authored-by: Tommaso <[email protected]>
Co-authored-by: Laurenz Rettig <[email protected]>
Co-authored-by: Carola Emminger <[email protected]>
Co-authored-by: kuehbachm <[email protected]>

.github/workflows/pylint.yml

@@ -17,6 +17,9 @@ jobs:
         python -m pip install --upgrade pip
         pip install nomad-lab==1.0.0 --extra-index-url https://gitlab.mpcdf.mpg.de/api/v4/projects/2187/packages/pypi/simple
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    - name: Install package
+      run: |
+        python -m pip install .
     - name: pycodestyle
       run: |
         python -m pycodestyle --ignore=E501,E701,E731 nexusparser tests

.vscode/settings.json

@@ -1,4 +1,5 @@
 {
+    "python.pythonPath": "/home/andrea/NOMAD/.pyenv/bin/python",
     "editor.rulers": [90],
     "editor.renderWhitespace": "all",
     "editor.tabSize": 4,
@@ -7,22 +8,22 @@
     },
     "files.trimTrailingWhitespace": true,
     "python.linting.pylintArgs": [
-        "--load-plugins=pylint_mongoengine",
+        "--load-plugins=pylint_mongoengine"
     ],
     "python.linting.pycodestylePath": "pycodestyle",
     "python.linting.pycodestyleEnabled": true,
     "python.linting.pycodestyleArgs": ["--ignore=E501,E701,E731"],
-    "python.linting.mypyEnabled": false,
+    "python.linting.mypyEnabled": true,
     "python.linting.mypyArgs": [
         "--ignore-missing-imports",
         "--follow-imports=silent",
         "--no-strict-optional"
     ],
-    "python.linting.pylintEnabled": false,
+    "python.linting.pylintEnabled": true,
     "python.linting.enabled": true,
     "python.testing.pytestArgs": [
         "tests"
     ],
     "python.testing.unittestEnabled": false,
     "python.testing.pytestEnabled": true
-}
+}

README.md

@@ -7,24 +7,37 @@ the required nomad-lab pypi package requires many dependencies with specific ver
 that might conflict with other libraries that you have installed. This was tested
 with Python 3.7.
 
+If you don't have Python 3.7 installed on your computer, follow these commands:
 ```
+sudo add-apt-repository ppa:deadsnakes/ppa
+sudo apt install python3.7 python3-dev libpython3.7-dev python-numpy
+
+```
+
+You can now install your virtual environment with python3.7 interpreter
+
+```
+mkdir <your-brand-new-folder>
+cd <your-brand-new-folder>
 pip install virtualenv
-virtualenv -p `which python3` .pyenv
+virtualenv --python=python3.7 .pyenv
 source .pyenv/bin/activate
 ```
 
 Simply install our pypi package with pip:
 ```
 pip install --upgrade pip
-pip install nomad-lab
+pip install nomad-lab==1.0.0 --extra-index-url https://gitlab.mpcdf.mpg.de/api/v4/projects/2187/packages/pypi/simple
 ```
 
 Clone this project (or fork and then clone the fork). Go into the cloned directly and
 directly run the parser from there:
 ```
 git clone https://github.com/nomad-coe/nomad-parser-nexus.git parser-nexus
 cd parser-nexus
-python -m nexusparser tests/data/nexus.out
+git submodule sync --recursive
+git submodule update --init --recursive
+
 ```
 
 There are also a basic test framework written in [pytest](https://docs.pytest.org/en/stable/).
@@ -55,14 +68,14 @@ This directory have several modules/files as following:
 ### **Tools**
 This module contains several tools that are necessary to read user-specific experimental data and parse them into NOMAD.
 
-**1. Read_nexus**
+**1. Read nexus**
 
 First go to the nexus tools directory:
 ```
 cd nexusparser/tools
 ```
 
-The module `read_nexus.py` reads HDF5 data file (i.e it is a user-specific data file, which contains numerical and metadata about the experiment. And it is formatted in NEXUS style.)
+The module `nexus.py` reads HDF5 data file (i.e it is a user-specific data file, which contains numerical and metadata about the experiment. And it is formatted in NEXUS style.)
 
 <span style="color:red">*The environmental variable called "NEXUS_DEF_PATH" should be set to the directory, which contains application definitions as XML files. If this environmental variable is not defined, then code will first clone a directory from GitHub, which contains application definitions.*</span>
 
@@ -71,13 +84,13 @@ To set environmental variable, you can do:
 export 'NEXUS_DEF_PATH'=<folder_path_that_contains_app_defs>
 ```
 
-***Testing read_nexus***
+***Testing nexus***
 
-Following example dataset can be used to test `read_nexus.py` module `tests/data/nexus_test_data/201805_WSe2_arpes.nxs`. This is angular resolved photoelectron spectroscopy (ARPES) dataset and it is formatted according to the [NXarpes application definition of NEXUS](https://manual.nexusformat.org/classes/applications/NXarpes.html#nxarpes). Run the following command to test the `read_nexus.py` using example ARPES dataset:
+Following example dataset can be used to test `nexus.py` module `tests/data/nexus_test_data/201805_WSe2_arpes.nxs`. This is angular resolved photoelectron spectroscopy (ARPES) dataset and it is formatted according to the [NXarpes application definition of NEXUS](https://manual.nexusformat.org/classes/applications/NXarpes.html#nxarpes). Run the following command to test the `nexus.py` using example ARPES dataset:
 ```
 python test_parser.py
 ```
-*You should get "Testing of read_nexus.py is SUCCESSFUL." message, if everything goes as expected!*
+*You should get "Testing of nexus.py is SUCCESSFUL." message, if everything goes as expected!*
 
 
 

nexusparser/__init__.py

@@ -1,3 +1,6 @@
+"""init file
+
+"""
 #
 # Copyright The NOMAD Authors.
 #
@@ -15,5 +18,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-
 from .parser import NexusParser

nexusparser/__main__.py

@@ -1,3 +1,6 @@
+"""main file of tools module
+
+"""
 #
 # Copyright The NOMAD Authors.
 #
@@ -27,6 +30,6 @@
 
 if __name__ == "__main__":
     configure_logging(console_log_level=logging.DEBUG)
-    archive = EntryArchive()
-    NexusParser().parse(sys.argv[1], archive, logging)
-    json.dump(archive.m_to_dict(), sys.stdout, indent=2)
+    ARCHIVE = EntryArchive()
+    NexusParser().parse(sys.argv[1], ARCHIVE, logging)
+    json.dump(ARCHIVE.m_to_dict(), sys.stdout, indent=2)

nexusparser/metainfo/__init__.py

@@ -1,3 +1,6 @@
+"""init file
+
+"""
 #
 # Copyright The NOMAD Authors.
 #
@@ -16,8 +19,3 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
-from nomad.metainfo import Environment
-
-from . import nexus
-
-m_env = Environment()