Adds support for reading igor files

[domna]

This converts igor data as collected at the FHI to nexus.

Install: Just install the plugin from this branch with pynx (may need a reinstall because of a new entry point)
Usage: dataconverter --reader igor_fhi --nxdl NXmpes -c your-config-file.json data-dir-with-ibw-files

[domna]

@rettigl The first base version of this reader is finished. It converts run it finds into a separate entry in the nexus file. I did not include the ibw testing data here, because I wasn't sure if it's fine if I upload them to github. But running the test locally is fine.

[rettigl]

I'll habe a look, thanks. I will look for some data to use as test data

[domna]

It does not parse all the metadata yet. It only constructs the data part (but I think this is the complex part). I have to see what is in the full ibw notes, but generally they can just be read out with @attrs:... (I'm currently at implementing this mechanism).

Currently, I get this warnings:

The data entry corresponding to /ENTRY[entry1]/title is required and hasn't been supplied by the reader.
The data entry corresponding to /ENTRY[entry1]/start_time is required and hasn't been supplied by the reader.
The required group, /ENTRY[entry1]/INSTRUMENT, hasn't been supplied.
The required group, /ENTRY[entry1]/SAMPLE, hasn't been supplied.
The data entry corresponding to /ENTRY[entry2]/title is required and hasn't been supplied by the reader.
The data entry corresponding to /ENTRY[entry2]/start_time is required and hasn't been supplied by the reader.
The required group, /ENTRY[entry2]/INSTRUMENT, hasn't been supplied.
The required group, /ENTRY[entry2]/SAMPLE, hasn't been supplied.
The data entry corresponding to /ENTRY[entry3]/title is required and hasn't been supplied by the reader.
The data entry corresponding to /ENTRY[entry3]/start_time is required and hasn't been supplied by the reader.
The required group, /ENTRY[entry3]/INSTRUMENT, hasn't been supplied.
The required group, /ENTRY[entry3]/SAMPLE, hasn't been supplied.
The data entry corresponding to /ENTRY[entry4]/title is required and hasn't been supplied by the reader.
The data entry corresponding to /ENTRY[entry4]/start_time is required and hasn't been supplied by the reader.
The required group, /ENTRY[entry4]/INSTRUMENT, hasn't been supplied.
The required group, /ENTRY[entry4]/SAMPLE, hasn't been supplied.
The data entry corresponding to /ENTRY[entry5]/title is required and hasn't been supplied by the reader.
The data entry corresponding to /ENTRY[entry5]/start_time is required and hasn't been supplied by the reader.
The required group, /ENTRY[entry5]/INSTRUMENT, hasn't been supplied.
The required group, /ENTRY[entry5]/SAMPLE, hasn't been supplied.

[domna]

I added now support that each note can be addressed with the @attrs keyword in the python file, e.g., @attrs:Ek to get the Ek value from the igor notes. With this you can construct your file and I would leave the association to you as I don't know the exact meaning of all of these parameters and the best place for you to store them. Additional data can be entered with a nomad eln (could also go through an additional json or yaml file, but this we would need to add to the extensions).

[rettigl]

This is what I get when trying it out (on python 3.8):

{
	"name": "TypeError",
	"message": "'key' is an invalid keyword argument for insort_right()",
	"stack": "---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 convert(
      2     input_file=glob(str(\"/mnt/topfloor/StaticARPES/Data/ARPES/2024/202406/20240611/\" + \"*.ibw\")),
      3     reader=\"igor_fhi\",
      4     nxdl=\"NXmpes\",
      5     output=\"./igor_test.nxs\",
      6     config_file=\"./data/igor/config_file.json\",
      7     skip_verify=False,
      8     ignore_undocumented=False,
      9 )

File /mnt/pcshare/users/Laurenz/AreaB/transformations_testing/pynxtools/src/pynxtools/dataconverter/convert.py:233, in convert(input_file, reader, nxdl, output, skip_verify, **kwargs)
    204 \"\"\"The conversion routine that takes the input parameters and calls the necessary functions.
    205 
    206 Parameters
   (...)
    228 None.
    229 \"\"\"
    231 nxdl_root, nxdl_f_path = helpers.get_nxdl_root_and_path(nxdl)
--> 233 data = transfer_data_into_template(
    234     input_file=input_file,
    235     reader=reader,
    236     nxdl_name=nxdl,
    237     nxdl_root=nxdl_root,
    238     skip_verify=skip_verify,
    239     **kwargs,
    240 )
    242 helpers.add_default_root_attributes(data=data, filename=os.path.basename(output))
    243 Writer(data=data, nxdl_f_path=nxdl_f_path, output_path=output).write()

File /mnt/pcshare/users/Laurenz/AreaB/transformations_testing/pynxtools/src/pynxtools/dataconverter/convert.py:174, in transfer_data_into_template(input_file, reader, nxdl_name, nxdl_root, skip_verify, **kwargs)
    171 else:
    172     fail = False
--> 174 data = data_reader().read(  # type: ignore[operator]
    175     template=Template(template), file_paths=input_file, **kwargs
    176 )
    177 entry_names = data.get_all_entry_names()
    178 for entry_name in entry_names:

File /mnt/pcshare/users/Laurenz/AreaB/transformations_testing/pynxtools/src/pynxtools/dataconverter/readers/multi/reader.py:322, in MultiFormatReader.read(self, template, file_paths, objects, **kwargs)
    319 template.update(self.setup_template())
    320 template.update(self.handle_objects(objects))
--> 322 self.post_process()
    323 if self.config_file is not None:
    324     template.update(
    325         parse_json_config(
    326             self.config_file, self.get_entry_names(), self.callbacks
    327         )
    328     )

File /mnt/pcshare/users/Laurenz/AreaB/transformations_testing/pynxtools-mpes/pynxtools_mpes/igor.py:177, in IgorReader.post_process(self)
    167 def post_process(self) -> None:
    168     \"\"\"
    169     Reads the igor binarywave files and returns a dictionary containing the data.
    170 
   (...)
    175         Dict[str, Any]: The dictionary containing the data.
    176     \"\"\"
--> 177     for scan_no, files in find_scan_sets(self.ibw_files).items():
    178         self.scan_nos.append(scan_no)
    179         waves = []

File /mnt/pcshare/users/Laurenz/AreaB/transformations_testing/pynxtools-mpes/pynxtools_mpes/igor.py:86, in find_scan_sets(filenames, pattern)
     84         if scan not in scan_sets:
     85             scan_sets[scan] = []
---> 86         insort(scan_sets[scan], fn, key=lambda fn: sort_key(fn, pattern))
     87 return scan_sets

TypeError: 'key' is an invalid keyword argument for insort_right()"
}

Apparently, the key keyword is only support for Python 3.10 and above.

[domna]

Apparently, the key keyword is only support for Python 3.10 and above.

I see. I wasn't aware of this. I changed it to list.sort(key=....) which is supported also for 3.8.

[rettigl]

This works to some extend now, and is a very good starting point. I need to think a little about how much of the details of our experiment should be in this reader, or if I rather move it to a different package.
Maybe it would be best to move all the igor-specific code into the main multiformat reader, add ibw to the general mpes reader, and have a separate package to handle the scan structure of our files.

[domna]

Maybe it would be best to move all the igor-specific code into the main multiformat reader, add ibw to the general mpes reader, and have a separate package to handle the scan structure of our files.

@lukaspie and I already discussed this but we decided against, because the igor files we had already were so different. Also the reading of these files mainly employ the specific file structure you have and not the actual reading of the ibw file (which is actually just an binarywave.load(...))