diff --git a/joss.06371/10.21105.joss.06371.crossref.xml b/joss.06371/10.21105.joss.06371.crossref.xml new file mode 100644 index 0000000000..080e1deb77 --- /dev/null +++ b/joss.06371/10.21105.joss.06371.crossref.xml @@ -0,0 +1,302 @@ + + + + 20240311T143534-10c1c0a757c4a43ea07e9613e559c70baca0a4e2 + 20240311143534 + + JOSS Admin + admin@theoj.org + + The Open Journal + + + + + Journal of Open Source Software + JOSS + 2475-9066 + + 10.21105/joss + https://joss.theoj.org + + + + + 03 + 2024 + + + 9 + + 95 + + + + NOMAD CAMELS: Configurable Application for +Measurements, Experiments and Laboratory Systems + + + + Alexander D. + Fuchs + https://orcid.org/0000-0003-1896-9242 + + + Johannes A. F. + Lehmeyer + https://orcid.org/0000-0003-2041-9987 + + + Heinz + Junkes + https://orcid.org/0000-0002-0218-4873 + + + Heiko B. + Weber + https://orcid.org/0000-0002-6403-9022 + + + Michael + Krieger + https://orcid.org/0000-0003-1480-9161 + + + + 03 + 11 + 2024 + + + 6371 + + + 10.21105/joss.06371 + + + http://creativecommons.org/licenses/by/4.0/ + http://creativecommons.org/licenses/by/4.0/ + http://creativecommons.org/licenses/by/4.0/ + + + + Software archive + 10.5281/zenodo.10793952 + + + GitHub review issue + https://github.com/openjournals/joss-reviews/issues/6371 + + + + 10.21105/joss.06371 + https://joss.theoj.org/papers/10.21105/joss.06371 + + + https://joss.theoj.org/papers/10.21105/joss.06371.pdf + + + + + + NOMAD: A distributed web-based platform for +managing materials science research data + Scheidgen + Journal of Open Source +Software + 90 + 8 + 10.21105/joss.05388 + 2475-9066 + 2023 + Scheidgen, M., Himanen, L., Ladines, +A. N., Sikter, D., Nakhaee, M., Fekete, Á., Chang, T., Golparvar, A., +Márquez, J. A., Brockhauser, S., Brückner, S., Ghiringhelli, L. M., +Dietrich, F., Lehmberg, D., Denell, T., Albino, A., Näsström, H., +Shabih, S., Dobener, F., … Draxl, C. (2023). NOMAD: A distributed +web-based platform for managing materials science research data. Journal +of Open Source Software, 8(90), 5388. +https://doi.org/10.21105/joss.05388 + + + FAIR Research Data With NOMAD: FAIRmat’s +Distributed, Schema-based Research-data Infrastructure to Harmonize RDM +in Materials Science + Scheidgen + Proceedings of the Conference on Research +Data Infrastructure + 1 + 10.52825/cordi.v1i.376 + 2941-296X + 2023 + Scheidgen, M., Brückner, S., +Brockhauser, S., Ghiringhelli, L. M., Dietrich, F., Mansour, A. E., +Márquez, J. A., Albrecht, M., Weber, H. B., Botti, S., Aeschlimann, M., +& Draxl, C. (2023). FAIR Research Data With NOMAD: FAIRmat’s +Distributed, Schema-based Research-data Infrastructure to Harmonize RDM +in Materials Science. Proceedings of the Conference on Research Data +Infrastructure, 1. +https://doi.org/10.52825/cordi.v1i.376 + + + NOMAD CAMELS documentation + Fuchs + 2023 + Fuchs, A. D., & Lehmeyer, J. A. +F. (2023). NOMAD CAMELS documentation. +https://fau-lap.github.io/NOMAD-CAMELS/ + + + PyMoDAQ + 2023 + PyMoDAQ. (2023). +https://pymodaq.cnrs.fr/ + + + PyVISA + 2023 + PyVISA. (2023). +https://pyvisa.readthedocs.io/ + + + FAIRmat NeXus proposal + 2023 + FAIRmat NeXus proposal. (2023). +https://fairmat-nfdi.github.io/nexus-fairmat-proposal/9636feecb79bb32b828b1a9804269573256d7696/fairmat-cover.html + + + DFG Positionspapier: Förderung von +Informationsinfrastrukturen für die Wissenschaft + 2018 + DFG Positionspapier: Förderung von +Informationsinfrastrukturen für die Wissenschaft. (2018). +https://www.dfg.de/download/pdf/foerderung/programme/lis/positionspapier_informationsinfrastrukturen.pdf + + + pip documentation V23.3.1 + 2023 + pip documentation V23.3.1. (2023). +https://pip.pypa.io/ + + + Instrument Control (iC) An Open-Source +Software to Automate Test Equipment + Pernstich + Journal of Research of the National Institute +of Standards and Technology + 117 + 10.6028/jres.117.010 + 1044-677X + 2012 + Pernstich, K. P. (2012). Instrument +Control (iC) An Open-Source Software to Automate Test Equipment. Journal +of Research of the National Institute of Standards and Technology, 117, +176–184. https://doi.org/10.6028/jres.117.010 + + + Bluesky Project + 2023 + Bluesky Project. (2023). +https://blueskyproject.io/ + + + Bluesky’s Ahead: A Multi-Facility +Collaboration for an a la Carte Software Project for Data Acquisition +and Management + Allan + Synchrotron Radiat. News + 3 + 32 + 10.1080/08940886.2019.1608121 + 2019 + Allan, D., Caswell, T., Campbell, S., +& Rakitin, M. (2019). Bluesky’s Ahead: A Multi-Facility +Collaboration for an a la Carte Software Project for Data Acquisition +and Management. Synchrotron Radiat. News, 32(3), 19–22. +https://doi.org/10.1080/08940886.2019.1608121 + + + EPICS - Experimental Physics and Industrial +Control System + 2023 + EPICS - Experimental Physics and +Industrial Control System. (2023). +https://epics-controls.org/ + + + CAMELS_drivers: instrument implementation for +CAMELS + 2023 + CAMELS_drivers: instrument +implementation for CAMELS. (2023). +https://github.com/FAU-LAP/CAMELS_drivers + + + SweepMe! - A multi-tool measurement +software + 2023 + SweepMe! - A multi-tool measurement +software. (2023). https://sweep-me.net/ + + + The NeXus data format + Könnecke + J. Appl. Crystallogr. + 1 + 48 + 10.1107/S1600576714027575 + 2015 + Könnecke, M., Akeroyd, F. A., +Bernstein, H. J., Brewster, A. S., Campbell, S. I., Clausen, B., +Cottrell, S., Hoffmann, J. U., Jemian, P. R., Männicke, D., Osborn, R., +Peterson, P. F., Richter, T., Suzuki, J., Watts, B., Wintersberger, E., +& Wuttke, J. (2015). The NeXus data format. J. Appl. Crystallogr., +48(1), 301–305. +https://doi.org/10.1107/S1600576714027575 + + + EPICS: A control system software +co-development success story + Knott + Nucl. Inst. Methods Phys. Res. +A + 1-2 + 352 + 10.1016/0168-9002(94)91577-6 + 1994 + Knott, M., Gurd, D., Lewis, S., & +Thuot, M. (1994). EPICS: A control system software co-development +success story. Nucl. Inst. Methods Phys. Res. A, 352(1-2), 486–491. +https://doi.org/10.1016/0168-9002(94)91577-6 + + + The FAIR Guiding Principles for scientific +data management and stewardship + Wilkinson + Sci. Data + 1 + 3 + 10.1038/sdata.2016.18 + 2016 + Wilkinson, M. D., Dumontier, M., +Aalbersberg, Ij. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., +Boiten, J. W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., +Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, +S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR Guiding +Principles for scientific data management and stewardship. Sci. Data, +3(1), 1–9. https://doi.org/10.1038/sdata.2016.18 + + + + + + diff --git a/joss.06371/10.21105.joss.06371.jats b/joss.06371/10.21105.joss.06371.jats new file mode 100644 index 0000000000..ed3837f5d5 --- /dev/null +++ b/joss.06371/10.21105.joss.06371.jats @@ -0,0 +1,669 @@ + + +
+ + + + +Journal of Open Source Software +JOSS + +2475-9066 + +Open Journals + + + +6371 +10.21105/joss.06371 + +NOMAD CAMELS: Configurable Application for Measurements, +Experiments and Laboratory Systems + + + +https://orcid.org/0000-0003-1896-9242 + +Fuchs +Alexander D. + + + + + +https://orcid.org/0000-0003-2041-9987 + +Lehmeyer +Johannes A. F. + + + + + +https://orcid.org/0000-0002-0218-4873 + +Junkes +Heinz + + + + +https://orcid.org/0000-0002-6403-9022 + +Weber +Heiko B. + + + + +https://orcid.org/0000-0003-1480-9161 + +Krieger +Michael + + +* + + + +Lehrstuhl für Angewandte Physik, Department Physik, +Friedrich-Alexander Universität Erlangen-Nürnberg, +Germany. + + + + +Physics Department and CSMB, Humboldt-Universität zu +Berlin, Berlin, Germany. + + + + +Fritz-Haber-Institut, Berlin, Germany. + + + + +* E-mail: + + +20 +11 +2023 + +9 +95 +6371 + +Authors of papers retain copyright and release the +work under a Creative Commons Attribution 4.0 International License (CC +BY 4.0) +2022 +The article authors + +Authors of papers retain copyright and release the work under +a Creative Commons Attribution 4.0 International License (CC BY +4.0) + + + +Python + + + + + + Summary +

NOMAD CAMELS (short: CAMELS) is a configurable, open-source + measurement software that records fully self-describing experimental + data. It has its origins in the field of experimental physics where a + wide variety of measurement instruments are used in frequently + changing experimental setups and measurement protocols. CAMELS + provides a graphical user interface (GUI) which allows the user to + configure experiments without the need of programming skills or deep + understanding of instrument communication. CAMELS translates + user-defined measurement protocols into stand-alone executable Python + code for full transparency of the actual measurement sequences. + Existing large-scale, distributed control systems, such as EPICS + (EPICS, + 2023) can be natively implemented. CAMELS is designed with a + focus on full recording of data and metadata. When shared with others, + data produced with CAMELS allow full understanding of the measurement + and the resulting data in accordance with the FAIR + (Findable, Accessible, + Interoperable and Re-usable) principles + (Wilkinson + et al., 2016).

+
+ + Statement of need +

Research data management has piqued greater and greater interest in + recent years. Today, research funding agencies demand sustainable + research data strategies. The key criterion is to create research data + following the FAIR principles and thereby improve world-wide + data-driven research + (DFG + Positionspapier, 2018). While one ingredient, + electronic lab notebooks (ELNs), are an important step towards FAIR + data, it is equally important to record the measurement data and + related metadata along FAIR principles as early as possible in the + research workflow.

+

In experimental physics many custom-built measurement setups are + controlled by very specific software written by individual + researchers. This results in a heterogeneous landscape of software + fragments for measurements written in many different languages and + with often incomplete documentation, making it almost impossible for + other researchers to extend existing code. The degree to which the + stored raw data is understandable varies greatly but is often + unintelligible even for researchers from the same lab. Important + metadata such as instrument settings or the actual measurement steps + performed to obtain the final raw data are rarely recorded, making it + virtually impossible to reproduce experiments. Therefore, the + documentation of experiments is incomplete, preventing FAIR research + data. Although there are some tools available to realise control of + arbitrary measurement instruments, such as SweepMe! + (SweepMe, + 2023), iC + (Pernstich, + 2012), and PyMoDAQ + (PyMoDAQ, + 2023), they are frequently not open-source or their data output + is not compliant with the FAIR principles.

+ +

Visualization of CAMELS functionality and workflow. + CAMELS connects directly with local instruments and/or large-scale + lab infrastructure running network protocols, e.g., EPICS. + Customizable measurements protocols are translated into Python code + and executed. The output is FAIR-compliant measurement data. +

+ +
+
+ + NOMAD CAMELS +

CAMELS is an open-source tool that automatically collects all + computer-accessible experimental metadata. It features a user-friendly + graphical interface that enables the creation and customization of + measurements without the need for programming knowledge. By default, + the data is stored in a structured HDF5 file format that closely + resembles the structure of the NeXus standard + (FAIRmat + NeXus Proposal, 2023; + Könnecke + et al., 2015). The final HDF5 file contains both the actual + measurement data and metadata in a single file, compliant to the FAIR + principles.

+

CAMELS is a stand-alone desktop application. It allows for direct + access to the NOMAD repository + (Scheidgen, + Brückner, et al., 2023; + Scheidgen, + Himanen, et al., 2023) or its on-premise installation called + NOMAD Oasis enabling direct linking to electronic lab + notebook entries. The user can for example connect measurements to + previous experiment workflows documented in NOMAD + ELNs. CAMELS can subsequently upload measurement results directly into + the ELN providing a simple and stream-lined data workflow.

+

CAMELS builds on bluesky + (Allan + et al., 2019; + Bluesky, + 2023) initially developed to control instruments at large-scale + research institutions using EPICS + (EPICS, + 2023; + Knott + et al., 1994). In CAMELS, bluesky is employed + to orchestrate the instrument communication either directly, e.g., via + PyVISA + (PyVISA, + 2023) or using network protocols. Existing lab infrastructure + controlled by EPICS is therefore immediately accessible. A schematic + overview of the functionality of CAMELS is displayed in + [fig:camels_overview].

+

CAMELS provides a comprehensive set of functionalities that can be + split into three primary components: instrument management, + measurement protocols, and manual controls.

+ +

(a) The instrument manager allows to + install and configure instrument drivers from the curated instrument + driver library or the user’s hard drive. (b) The + measurement protocol editor allows users to configure arbitrary + measurement sequences. +

+ +
+ + Instrument Management +

Scientific instruments can be added to CAMELS in two ways: The + first involves the instrument manager (c.f. + [fig:manager_protocols]a) + to add instruments from the official curated driver repository + (CAMELS-drivers, + 2023). These drivers are installed into the Python + environment via pip + (Pip, + 2023) with each driver being packaged individually.

+

The second way is to add self-built drivers by creating the + necessary files locally and placing these in the directory specified + in the CAMELS settings. To facilitate this process CAMELS provides a + driver builder that automatically generates the + essential structure and boilerplate code. As CAMELS is an + open-source project developed by and for the community, users are + encouraged to contribute to the driver library by creating pull + requests for new drivers on the GitHub repository + (CAMELS-drivers, + 2023).

+

In general, a CAMELS driver comprises two files: One containing + the hardware interface communication, the other one defining the + available instrument settings. Data communication to instruments is + handled via channels that can be set and/or read; + they correspond to an instrument’s individual functionality or + physical property.

+
+ + Measurement protocols +

In CAMELS a measurement protocol is a + distinctive sequence of individual steps including setting and + reading instrument channels (see + [fig:manager_protocols]b), + loops, conditional execution, running sub-protocols, PID control, + etc. This yields a measurement in a ‘recipe-style’ format, where the + next step is usually executed after the successful completion of the + preceding step. Asynchronous data acquisition is supported.

+

CAMELS translates the protocol created in the GUI into a Python + script, which is then executed. The script can be viewed, run + independently of CAMELS, and modified if required. CAMELS protocols + and settings can be stored and shared with colleagues enabling easy + repeatability of experiments.

+
+ + Manual controls +

Certain scientific instruments require manual control before + starting predefined measurement routines, e.g., adjusting stages, + controlling temperature, valves, pumps, etc. In CAMELS this is + achieved through the manual controls functionality + which can be applied to any writable instrument channel.

+
+ + Data output + +

CAMELS stores measurement data together with rich + metadata collected automatically into a structured HDF5 file by + default. This includes (a) a human readable + measurement protocol summary and (b) the executable + Python script that was used to actually record the data. This + allows others to understand the data acquisition and to reproduce + the experiment. +

+ +
+

After executing the measurement protocol, the time-stamped data + is by default saved to an HDF5 file with a structure similar to the + NeXus standard + (Könnecke + et al., 2015). Data can also be exported in CSV format with + the metadata exported in JSON.

+

The stored data can be divided into distinct sections:

+ + +

Time-stamped raw data obtained during the execution of the + measurement protocol.

+
+ +

Instrument settings.

+
+ +

Human-readable summary of the measurement protocol + information (see + [fig:h5_data]a).

+
+ +

Complete Python script that recorded the data (see + [fig:h5_data]b) + as well as information on the Python environment, i.e., a list + of used packages and versions.

+
+ +

User-defined metadata, e.g., sample and user information.

+
+
+
+ + Documentation +

In-depth documentation and guides for installing, using and + troubleshooting can be found on the + CAMELS + documentation webpage + (Fuchs + & Lehmeyer, 2023).

+
+
+ + Acknowledgements +

We thank Patrick Oppermann (Fritz-Haber-Institut der + Max-Planck-Gesellschaft) for valuable discussions.

+

NOMAD CAMELS is being developed within the NFDI consortium + FAIRmat funded by the Deutsche Forschungsgemeinschaft + “DFG, German Research Foundation”, project 460197019.

+
+ + + + + + + ScheidgenMarkus + HimanenLauri + LadinesAlvin Noe + SikterDavid + NakhaeeMohammad + FeketeÁdám + ChangTheodore + GolparvarAmir + MárquezJosé A. + BrockhauserSandor + BrücknerSebastian + GhiringhelliLuca M. + DietrichFelix + LehmbergDaniel + DenellThea + AlbinoAndrea + NäsströmHampus + ShabihSherjeel + DobenerFlorian + KühbachMarkus + MozumderRubel + RudzinskiJoseph F. + DaelmanNathan + PizarroJosé M. + KubanMartin + SalazarCuauhtemoc + OndračkaPavel + BungartzHans-Joachim + DraxlClaudia + + NOMAD: A distributed web-based platform for managing materials science research data + Journal of Open Source Software + 202310 + 20231103 + 8 + 90 + 2475-9066 + 10.21105/joss.05388 + 5388 + + + + + + + ScheidgenMarkus + BrücknerSebastian + BrockhauserSandor + GhiringhelliLuca M. + DietrichFelix + MansourAhmed E. + MárquezJosé A. + AlbrechtMartin + WeberHeiko B. + BottiSilvana + AeschlimannMartin + DraxlClaudia + + FAIR Research Data With NOMAD: FAIRmat’s Distributed, Schema-based Research-data Infrastructure to Harmonize RDM in Materials Science + Proceedings of the Conference on Research Data Infrastructure + 202309 + 20231103 + 1 + 2941-296X + 10.52825/cordi.v1i.376 + + + + + + FuchsAlexander D. + LehmeyerJohannes A. F. + + NOMAD CAMELS documentation + 2023 + 20231031 + https://fau-lap.github.io/NOMAD-CAMELS/ + + + + + PyMoDAQ + 2023 + 20231030 + https://pymodaq.cnrs.fr/ + + + + + PyVISA + 2023 + 20231030 + https://pyvisa.readthedocs.io/ + + + + + FAIRmat NeXus proposal + 2023 + 20231030 + https://fairmat-nfdi.github.io/nexus-fairmat-proposal/9636feecb79bb32b828b1a9804269573256d7696/fairmat-cover.html + + + + + DFG Positionspapier: Förderung von Informationsinfrastrukturen für die Wissenschaft + 2018 + 20231030 + https://www.dfg.de/download/pdf/foerderung/programme/lis/positionspapier_informationsinfrastrukturen.pdf + + + + + pip documentation V23.3.1 + 2023 + 20231030 + https://pip.pypa.io/ + + + + + + PernstichK. P. + + Instrument Control (iC) An Open-Source Software to Automate Test Equipment + Journal of Research of the National Institute of Standards and Technology + 201205 + 20231030 + 117 + 1044-677X + 10.6028/jres.117.010 + 26900522 + 176 + 184 + + + + + Bluesky Project + 2023 + 20231005 + https://blueskyproject.io/ + + + + + + AllanDaniel + CaswellThomas + CampbellStuart + RakitinMaksim + + Bluesky’s Ahead: A Multi-Facility Collaboration for an a la Carte Software Project for Data Acquisition and Management + Synchrotron Radiat. News + Taylor; Francis Ltd. + 201905 + 32 + 3 + https://www.tandfonline.com/doi/abs/10.1080/08940886.2019.1608121 + 10.1080/08940886.2019.1608121 + 19 + 22 + + + + + EPICS - Experimental Physics and Industrial Control System + 2023 + 20231005 + https://epics-controls.org/ + + + + + CAMELS_drivers: instrument implementation for CAMELS + 2023 + 20230928 + https://github.com/FAU-LAP/CAMELS_drivers + + + + + SweepMe! - A multi-tool measurement software + 2023 + 20230928 + https://sweep-me.net/ + + + + + + KönneckeMark + AkeroydFrederick A. + BernsteinHerbert J. + BrewsterAaron S. + CampbellStuart I. + ClausenBjörn + CottrellStephen + HoffmannJens Uwe + JemianPete R. + MännickeDavid + OsbornRaymond + PetersonPeter F. + RichterTobias + SuzukiJiro + WattsBenjamin + WintersbergerEugen + WuttkeJoachim + + The NeXus data format + J. Appl. Crystallogr. + International Union of Crystallography + 201501 + 48 + 1 + http://scripts.iucr.org/cgi-bin/paper?po5029 + 10.1107/S1600576714027575 + 301 + 305 + + + + + + KnottM. + GurdD. + LewisS. + ThuotM. + + EPICS: A control system software co-development success story + Nucl. Inst. Methods Phys. Res. A + North-Holland + 199412 + 352 + 1-2 + 10.1016/0168-9002(94)91577-6 + 486 + 491 + + + + + + WilkinsonMark D. + DumontierMichel + AalbersbergIJsbrand Jan + AppletonGabrielle + AxtonMyles + BaakArie + BlombergNiklas + BoitenJan Willem + da Silva SantosLuiz Bonino + BournePhilip E. + BouwmanJildau + BrookesAnthony J. + ClarkTim + CrosasMercè + DilloIngrid + DumonOlivier + EdmundsScott + EveloChris T. + FinkersRichard + Gonzalez-BeltranAlejandra + GrayAlasdair J. G. + GrothPaul + GobleCarole + GretheJeffrey S. + HeringaJaap + HoenPeter A. C. t + HooftRob + KuhnTobias + KokRuben + KokJoost + LusherScott J. + MartoneMaryann E. + MonsAlbert + PackerAbel L. + PerssonBengt + Rocca-SerraPhilippe + RoosMarco + SchaikRene van + SansoneSusanna Assunta + SchultesErik + SengstagThierry + SlaterTed + StrawnGeorge + SwertzMorris A. + ThompsonMark + Van Der LeiJohan + Van MulligenErik + VelteropJan + WaagmeesterAndra + WittenburgPeter + WolstencroftKatherine + ZhaoJun + MonsBarend + + The FAIR Guiding Principles for scientific data management and stewardship + Sci. Data + Nature Publishing Group + 201603 + 3 + 1 + https://www.nature.com/articles/sdata201618 + 10.1038/sdata.2016.18 + 26978244 + 1 + 9 + + + + +
diff --git a/joss.06371/10.21105.joss.06371.pdf b/joss.06371/10.21105.joss.06371.pdf new file mode 100644 index 0000000000..cdef474712 Binary files /dev/null and b/joss.06371/10.21105.joss.06371.pdf differ diff --git a/joss.06371/media/pictures/Bild_CAMELS.png b/joss.06371/media/pictures/Bild_CAMELS.png new file mode 100644 index 0000000000..4ed3a5a967 Binary files /dev/null and b/joss.06371/media/pictures/Bild_CAMELS.png differ diff --git a/joss.06371/media/pictures/CAMELS_manager_protocol.png b/joss.06371/media/pictures/CAMELS_manager_protocol.png new file mode 100644 index 0000000000..5e969388c7 Binary files /dev/null and b/joss.06371/media/pictures/CAMELS_manager_protocol.png differ diff --git a/joss.06371/media/pictures/h5_data.png b/joss.06371/media/pictures/h5_data.png new file mode 100644 index 0000000000..d97f55cf6f Binary files /dev/null and b/joss.06371/media/pictures/h5_data.png differ