Code

EM refactoring / EMv2#193

Code

Closed

mkuehbach wants to merge 85 commits intomasterFAIRmat-NFDI/pynxtools:masterfrom em_refactoringFAIRmat-NFDI/pynxtools:em_refactoringCopy head branch name to clipboard

+10,607-6,692

Commits

Commits on Aug 15, 2023

• Starting a refactoring of EM parsers using the refactored EM appdef and base class definitions from the base_class_templates nexus_definitions branch

  

  markus.kuehbach

  committed
  52c2c51

• Using also the forwarded converter NXDL files from the base_class_templates branch

  

  markus.kuehbach

  committed
  1638344

• Workpackage

  

  markus.kuehbach

  committed
  39d6a19

Commits on Aug 16, 2023

• Store initial work that it does not get lost but this is not even a draft

  

  markus.kuehbach

  committed
  79709b3

Commits on Aug 17, 2023

• dataconverter, added and tested successfully an option to append to an already existent NeXus/HDF5 file

  

  markus.kuehbach

  committed
  71371c7

• Added control logic to instruct the dataconverter to create the output file based on an existent input-file to which then the template data are appended. This functionality is useful in cases when …

  Show description for e1bb68b
  

  markus.kuehbach

  committed
  e1bb68b

Commits on Aug 18, 2023

• Implemented nxs_mtex subparser, default plot annotator, and NeXus root decorator

  

  markus.kuehbach

  committed
  6ddf56e

• Updated nxdefs to use latest refactored NXem appdef

  

  markus.kuehbach

  committed
  5dd2c66

• Added logic and code for annotating original references and atom_types from phase_names based on a table composed from an EM domain expert

  

  markus.kuehbach

  committed
  be663d2

• Returning an intermediate result for now, but the verification with fairmat is not for some reason completely bypassed, some silent assumptions in nexus and the verification code?

  

  markus.kuehbach

  committed
  d20e49e

Commits on Oct 19, 2023

• Prepped HDF5 parsing using subparsers

  

  markus.kuehbach

  committed
  d5ae63f

Commits on Oct 22, 2023

• Refactoring and implementation for normalization oxford, bruker, apex and tested

  

  markus.kuehbach

  committed
  9c708da

Commits on Oct 23, 2023

• Bug fixing, implementation for normalization old edax, community hebsd, and tested

  

  markus.kuehbach

  committed
  4076096

• Implemented xmap mapping, runthrough tests with all datasets in HDF5 family files, bugfixing

  

  markus.kuehbach

  committed
  f947f9d

Commits on Oct 24, 2023

• Implemented phase-specific plots, bugfixing

  

  markus.kuehbach

  committed
  ba45b67

• Runthrough with all HDF5 examples from the community was successful but the calibration of ipf and roi axis coordinates uses pixels currently but should use pixel times scan unit, next steps: i) im…

  Show description for fea7842
  

  markus.kuehbach

  committed
  fea7842

Commits on Oct 25, 2023

• Bugfix proper reading and handling of scan positions into calibrated scan positions

  

  markus.kuehbach

  committed
  0a2307f

Commits on Oct 26, 2023

• Added EBSD map downsampling and bugfixing of non working AM example, this completes the implementation of the tech partner HDF5 parsers for EBSD, next step linting and adding EDS reader (for IKZ, etc)

  

  markus.kuehbach

  committed
  1459b8c

Commits on Nov 7, 2023

• Designed logic to probe if DREAM3D instance has relevant data to be mapped on NXem

  

  mkuehbach

  committed
  acf860d

• Tested auto-detection of relevant group to harvest 3D EBSD data from and ran successfully on all datasets, 3/13 contained relevant content

  

  mkuehbach

  committed
  e16d8ea

Commits on Nov 8, 2023

• Parsing of content from three different DREAM3D cases working, ROI code implemented but ROI code does not populate template, implement IPF computation

  

  mkuehbach

  committed
  08a001a

Commits on Nov 9, 2023

• Fixed ROI overview for all examples, resolved successfully now, IPF next to complete use case 3D EBSD

  

  mkuehbach

  committed
  0c118b5

• DREAM3D to pyxem phase id management, 4d to 2d array reshaping to bypass xmaps, and H5Web refactoring done and working, 3D IPF maps is the only thing remaining for the DREAM3D example to work on th…

  Show description for 5be172a
  

  mkuehbach

  committed
  5be172a

• Implemented IPF color mapping without using an xmap but there is still a bug with the zeroth dimension of the results array, IPF color map key renders correctly though

  

  mkuehbach

  committed
  8e1af84

• 3D EBSD working

  

  mkuehbach

  committed
  33688c1

Commits on Dec 4, 2023

• Potentially breaking refactoring from the train-ride back from the FAIRmat user/project meeting but preparing the removal of the crystal xmap also for the 2D cases

  

  mkuehbach

  committed
  dcb0cec

• Update definitions to use fairmat 615ff37 which contains the merged in base_class_templates branch

  

  mkuehbach

  committed
  96fb1c4

• Manual merging in of fairmat 8620c4e into the em_refactoring branch to prepare with the resolving of possible merge conflicts

  

  mkuehbach

  committed
  7e1cfd9

• Revert nxdefs to 9998376

  

  mkuehbach

  committed
  460e49e

• Removed nyaml2nxdl and related tools and test to manually sync with pynxtools/master cd54fde

  

  mkuehbach

  committed
  abc99b6

• Merged fairmat in

  

  mkuehbach

  committed
  58ca8ca

Commits on Dec 6, 2023

• Rename nxs_hfive parser to signify that the difference with the nxs_ subparsers is not in that input comes from HDF5 files but that the backend code to compute harmonized quantities is pyxem or mte…

  Show description for 8540700
  atomprobe-tc

  committed
  8540700

• Testing pyxem workflow and the large 3D EBSD dataset which takes about 48GB RAM peak

  atomprobe-tc

  committed
  71a6a56

• Started the refactoring to discretize always all point cloud data which are not collected on a square grid that is smaller than the maximum possible extent supported by h5web, tested with use case …

  Show description for 97fcaa1
  atomprobe-tc

  committed
  97fcaa1

Commits on Dec 7, 2023

• Fixed bug for 207_2081

  atomprobe-tc

  committed
  19dd554

• Implemented xmap-free twod IPF rendering function tested successfully on 207_2081

  atomprobe-tc

  committed
  23c28d4

• Remove code using now obsolete xmap

  atomprobe-tc

  committed
  c2ce3a0

• Proliferate tiling and phase_id, scan_point consistency for other parsers

  atomprobe-tc

  committed
  1824163

• Fixed one of two failing tests, mpes test still fails, pycodestyling

  atomprobe-tc

  committed
  2090418

• pylinting, mypy, removal of unnecessary imports

  atomprobe-tc

  committed
  ff5cfa0

• Passing pycodestyle, pylinting, mypy now locally using py3.10.13 conda

  atomprobe-tc

  committed
  6cde0e9

• Successful run-through of all pyxem-processed examples (2D/3D) none of them failing, maximum RAM peaking at 2.7GB for largest 2D example 229_2096.oh5 with 20.3mio scan points and 23GB for largest 3…

  Show description for 013c59c
  atomprobe-tc

  committed
  013c59c

• Manual rebase against master 0c69581 to assure converter and nexus code is the same as 0c69581 master, it now is

  atomprobe-tc

  committed
  cb17fde

• Initial steps generic reader for images (TIF) to start with for IKZ (Albrecht, Brückner)

  atomprobe-tc

  committed
  f949e3b

Commits on Dec 8, 2023

• Decode tech partner magic

  atomprobe-tc

  committed
  00f4556

• Added working skeleton code for the TFS-specific TIFF parser, which fishes successfully metadata and converts into standardized python dict, ods document added to collect IKZ feedback how to map sp…

  Show description for 020dea0
  atomprobe-tc

  committed
  020dea0

Commits on Dec 11, 2023

• Added initial version for the configuration, i.e. concept mapping table to harmonize and map instances of TFS/FEI concepts on NeXus concepts, using a few examples which are possibly of interest to …

  Show description for 981e3c2
  atomprobe-tc

  committed
  981e3c2

• image_tiff_tfs: generalized logic to read in all TFS/FEI metadata

  atomprobe-tc

  committed
  df72956

• Fixed error in floating point value interpretation logic, added writing of NXimage_r_set instance based on TIFF content

  atomprobe-tc

  committed
  345d8e1

• Made also now the mapping of TFS vocabulary to NeXus work, awaiting now feedback from IKZ to implement mapping of further quantities, test NeXus file writes successfully, next step refactor em_spct…

  Show description for 99d55ba
  atomprobe-tc

  committed
  99d55ba

• linting, styling, mypying

  atomprobe-tc

  committed
  5aa9977

Commits on Dec 13, 2023

• Added rosettasciio, added skeleton for refactoring EDS, EELS, ADF using rosettasciio, collected feedback from the discussion with Robert Kernke (IKZ), next steps i) complete mapping of all detector…

  Show description for 49fb0a3
  atomprobe-tc

  committed
  49fb0a3

Commits on Dec 14, 2023

• Added prioritization level 1 and 2 quantities for all five imaging modes of the IKZ Apreo, tested generation of NeXus files for all five test images from Robert Kernke successfully, next step conti…

  Show description for 2ae1cb2
  atomprobe-tc

  committed
  2ae1cb2

Commits on Dec 15, 2023

• Added code to read EDAX's FOVIMAGE their cumbersome SPD from APEX files in preparation to normalize into NeXus concepts NXimage_r_set and NXspectrum_set instance that are required for EDS

  atomprobe-tc

  committed
  76d25fe

Commits on Dec 18, 2023

• Added skeleton for collection of image files from the Reduction of FeOx example from AXON Studio 10.4.4.1 aka Protochips PNG file collection reader

  atomprobe-tc

  committed
  d471f03

• Add xmltodict to flatten xml to Python dictionaries

  atomprobe-tc

  committed
  eccccb5

• Not yet finished code to read metadata but includes already working name mangling for AXON jargon, next steps i) clean parse_and_normalize, ii) populate template with metadata, iii) populate templa…

  Show description for a065978
  atomprobe-tc

  committed
  a065978

Commits on Dec 19, 2023

• Fixed bug, refactored get_metadata, added and tested successfully hashing of files inside zip, linting, styling, myping

  atomprobe-tc

  committed
  97c7f8a

• Removed deprecated zipfile37 in all remaining places using zipfile instead

  atomprobe-tc

  committed
  4c113ca

• Working parsing and mapping of some Protochips/AXONStudio metadata on NeXus

  atomprobe-tc

  committed
  9d3b8e7

• Functionally working version of the collection-of-PNG-files from Protochips AXON Studio FeOx Reduction example, missing functionalities and remaining issues with this parser, detector is written ou…

  Show description for a729d1f
  atomprobe-tc

  committed
  a729d1f

• Started refactoring of the em_nion parser to the so that zip-compressed project directories can be inspected for their content, next steps i) find how to get swift version, ii) polish ndata, hfive,…

  Show description for d1ecede
  atomprobe-tc

  committed
  d1ecede

• Pylinting, styling, myping

  atomprobe-tc

  committed
  0ec6243

Commits on Dec 20, 2023

• Found possible metadata quantity to decipher used nionswift version from, fixed reporting style to perform run-through analysis of all experiments performed in C. Koch's group

  atomprobe-tc

  committed
  5ea8824

Commits on Jan 5, 2024

• Fixed and confirmed that HdfFiveBaseParser can be used to analyze structure of any HDF5 file, tested for jlaehnemanns InGaN.edaxh5

  atomprobe-tc

  committed
  7f9cf4e

Commits on Jan 11, 2024

• Continuing on the spctrscpy examples, local dev directory tidied up

  atomprobe-tc

  committed
  f28abcf

• Further tidying up

  atomprobe-tc

  committed
  69e738f

• Reorganize location of tests, etc.

  atomprobe-tc

  committed
  ad768bc

Commits on Jan 12, 2024

• Reading and normalizing FOV, SPC, SPD, and LSD, next steps: i) add elementmap

  atomprobe-tc

  committed
  518a706

• Added other relevant conceptual groups from the three APEX examples shared VInP_108_L2 (IKZ Brückner, Bergmann), GeSi (IKZ Kernke), InP (PDI, laehnemann)

  atomprobe-tc

  committed
  baa02a6

• Added xraydb to get all X-ray emission lines

  atomprobe-tc

  committed
  bd4edec

Commits on Jan 13, 2024

• Tested parsing on IKZ and PDI examples

  atomprobe-tc

  committed
  712e384

Commits on Jan 15, 2024

• Started writing of normalized content to template, fov and h5web, and xray line identification

  atomprobe-tc

  committed
  bc2dc04

• Added plotting of spectrum

  atomprobe-tc

  committed
  4ae24cc

• Added simple id management for objects written to HDF5, bugfixes for pdi/InGe, ikz/AlGaO, and ikz/GeSi example with EBSD maps also working

  atomprobe-tc

  committed
  458e129

Commits on Jan 16, 2024

• Fixed VInP reading and 2D EDS element maps showing up now, started implementation of Velox EMD parser v2

  atomprobe-tc

  committed
  bfcd666

• Styling, linting, mypy

  atomprobe-tc

  committed
  1b2bd49

• Minor edit batch files

  atomprobe-tc

  committed
  fb0a996

Commits on Jan 17, 2024

• Edit to make version a child of definitions

  atomprobe-tc

  committed
  4a9de81

• Added first version of a imaging mode case distinction logic, tested imgs, adf, and ceta imaging modes. NeXus files were generated successfully but weird h5web display error coming up within ipynb,…

  Show description for 3892c70
  atomprobe-tc

  committed
  3892c70

• Added eds_map tested FHI Rohner emd file

  atomprobe-tc

  committed
  020225d

Commits on Jan 18, 2024

• Added support for y, x, energy, and energy EDS spectra, all content from FHI C. Rohner's example is parsed successfully, next steps: i) add Velox metadata schema version, ii) add microscope metadat…

  Show description for 1948a47
  atomprobe-tc

  committed
  1948a47

• Added i) schema version, started ii) mapping table for microscope metadata

  atomprobe-tc

  committed
  c185901

Commits on Jan 19, 2024

• A working version for parsing some of the Velox/FEI metadata and map them on NeXus

  atomprobe-tc

  committed
  1bd6c59

Commits on Jan 23, 2024

• Hooked in the correct NeXus definitions branch to use for the em_refactoring feature branch

  atomprobe-tc

  committed
  84b473d