diff --git a/joss.06296/10.21105.joss.06296.jats b/joss.06296/10.21105.joss.06296.jats new file mode 100644 index 0000000000..79dec074d8 --- /dev/null +++ b/joss.06296/10.21105.joss.06296.jats @@ -0,0 +1,787 @@ + + +
+ + + + +Journal of Open Source Software +JOSS + +2475-9066 + +Open Journals + + + +6296 +10.21105/joss.06296 + +Paicos: A Python package for analysis of (cosmological) +simulations performed with Arepo + + + +https://orcid.org/0000-0003-0466-603X + +Berlok +Thomas + + + +* + + +https://orcid.org/0009-0007-9039-294X + +Jlassi +Léna + + + + +https://orcid.org/0000-0001-8778-7587 + +Puchwein +Ewald + + + + +https://orcid.org/0000-0002-9422-8684 + +Haugbølle +Troels + + + + + +Niels Bohr Institute, University of Copenhagen, +Denmark + + + + +Leibniz-Institut für Astrophysik Potsdam, +Germany + + + + +* E-mail: + + +31 +1 +2024 + +9 +96 +6296 + +Authors of papers retain copyright and release the +work under a Creative Commons Attribution 4.0 International License (CC +BY 4.0) +2022 +The article authors + +Authors of papers retain copyright and release the work under +a Creative Commons Attribution 4.0 International License (CC BY +4.0) + + + +Python +astronomy + + + + + + Summary +

Cosmological simulations evolve dark matter and baryons subject to + gravitational and hydrodynamic forces + (Vogelsberger + et al., 2020). The simulations start at high redshift and + capture hierarchical structure formation where small structures form + first and later assemble to larger structures + (Springel + et al., 2005). The Arepo code is a versatile finite-volume code + which can solve the magnetohydrodynamic equations on an unstructured + Voronoi mesh in a cosmologically comoving frame + (Springel, + 2010; + Weinberger + et al., 2020).

+

Here we present Paicos, a new object-oriented Python package for + analyzing simulations performed with Arepo. Paicos strives to reduce + the learning curve for students and researchers getting started with + Arepo simulations. As such, Paicos includes many examples in the form + of Python scripts and Jupyter notebooks + (Kluyver + et al., 2016) as well as an online documentation describing the + installation procedure and recommended first steps.

+

Paicos’ main features are automatic handling of cosmological and + physical units, computation of derived variables, 2D visualization + (slices and projections), 1D and 2D histograms, and easy saving and + loading of derived data including units and all the relevant + metadata.

+

Paicos relies heavily on well-established open source software + libraries such as NumPy + (Harris + et al., 2020; + van + der Walt et al., 2011), Scipy + (Virtanen + et al., 2020), h5py + (Collette, + 2013), Cython + (Behnel + et al., 2011) and astropy + (Astropy + Collaboration, 2013) and contains features for interactive data + analysis inside Ipython terminals + (Perez + & Granger, 2007) and Jupyter notebooks + (Kluyver + et al., 2016), e.g., tab completion of data keywords and Latex + rendering of data units. Paicos also contains a number of tests that + are automated using pytest + (Krekel + et al., 2004), CircleCi and GitHub workflows.

+ + + Statement of need +

The Arepo code stores its data output as HDF5 files, which can + easily be loaded as NumPy arrays using h5py. However, data + visualization of the unstructured mesh used in Arepo is non-trivial + and keeping track of the units used in the data outputs can also be a + tedious task. The general purpose visualization and data analysis + software package yt1 + (Turk + et al., 2011) and the visualization package py-sphviewer + (Benitez-Llambay, + 2015) are both able to perform visualizations of Arepo + simulations. Paicos provides an alternative, which is specifically + tailored to Arepo simulations. It is in this regard similar to + swiftsimio + (Borrow + & Borrisov, 2020), which was developed specifically for + SWIFT simulations + (Schaller + & others, 2023).

+

We have developed Paicos because we identified a need for an + analysis code that simultaneously fulfills the following requirements: + 1) is specifically written for analysis of Arepo simulations + discretized on a Voronoi mesh 2) provides safeguards against errors + related to conversions of cosmological and physical units 3) + facilitates working with large data sets by supporting the saving and + loading of reduced/derived data including units and metadata 4) + contains enough functionality to be useful for practical research + tasks while still being light-weight and well-documented enough that + it can be installed, used and understood by a junior researcher with + little or no assistance.

+ + + Overview of key Paicos features +

The key Paicos features, which were implemented to fulfill the + above requirements, are as follows:

+ + +

Functionality for reading cell/particle and group/subhalo + catalog data as saved by Arepo in the HDF5 format.

+ + +

Unit-handling via astropy with additional support for the + + + a + and + + h + factors that are used in cosmological + simulations.2

+ + +

Functionality for automatically obtaining derived variables + from the variables present in the Arepo snapshot (e.g. the cell + volume, temperature, or magnetic field strength). Within Jupyter + notebooks or an Ipython terminal: Tab completion to show which + variables are available for a given data group (either by directly + loading from the HDF5-file or by automatically calculating a + derived variable).

+ + +

Functionality for saving and loading data including + cosmological and physical units in HDF5-files that automatically + include all metadata from the original Arepo snapshot (i.e. the + Header, Config and Param groups).

+ + +

Functionality for creating slices and projections and saving + them in a format that additionally includes information about the + size, center and orientation of the image.

+ + +

Functionality for creating 1D and 2D histograms (e.g. radial + profiles and + + ρ- + + T + phase-space plots) of large data sets (using OpenMP-enabled + Cython).

+ + +

Functionality for user customization, e.g. adding new units or + functions for computing derived variables.

+ + +

Functionality for selecting only a part of a snapshot for + analysis and for storing this selection as a new reduced + snapshot.

+ + +

Finally, we also make public GPU implementations of projection + functionalities, which are much faster than the OpenMP-parallel CPU + implementations described above. The GPU implementations include a + mass-conserving SPH-like projection and a ray tracing implementation + using a bounding volume hierarchy (BVH) in the form of a binary radix + tree for nearest neighbor searches. These GPU implementations use + Numba + (Lam + et al., 2015) and CuPy + (Okuta + et al., 2017) and require that the user has a CUDA-enabled GPU. + Our binary tree implementation follows the the GPU-optimized + tree-construction algorithm described in + (Karras, + 2012) and the implementation of it found in the publicly + available Cornerstone Octree GPU-library + (Keller + et al., 2023). Using our SPH-like implementation on an + Nvidia + A100 GPU, it takes less than a second to project 100 mio. + particles onto an image plane with + + 40962 + pixels. This speed enables interactive data exploration. We provide an + example IPython widget illustrating this feature. Finally, we note + that the returned images include physical units and can be used for + scientific analysis.

+

Paicos is hosted on GitHub at https://github.com/tberlok/paicos. We + strongly encourage contributions to Paicos by opening issues and + submitting pull requests.

+ + + Acknowledgements +

We thank the JOSS referees, Kyle Oman and Terrence Tricco, and the + editor, Josh Borrow, for constructive feedback that greatly helped + improve Paicos. We are grateful to Christoph Pfrommer for support and + advice. We thank Rüdiger Pakmor, Rosie Talbot and Timon Thomas for + useful discussions as well as Matthias Weber, Lorenzo Maria Perrone, + Arne Trabert and Joseph Whittingham for beta-testing Paicos. We are + grateful to Volker Springel for making the Arepo code available and to + the main developers of arepo-snap-util (Federico Marinacci and Rüdiger + Pakmor), a non-public code which we have used for comparison of + projections and slices. TB gratefully acknowledges funding from the + European Union’s Horizon Europe research and innovation programme + under the Marie Skłodowska-Curie grant agreement No 101106080. LJ + acknowledges support by the German Science Foundation (DFG) under + grant “DFG Research Unit FOR 5195 – Relativistic Jets in Active + Galaxies”. LJ and TB acknowledge support by the European Research + Council under ERC-AdG grant PICOGAL-101019746. The authors gratefully + acknowledge the Gauss Centre for Supercomputing e.V. + (www.gauss-centre.eu) for funding this project by providing computing + time on the GCS Supercomputer SuperMUC at Leibniz Supercomputing + Centre (www.lrz.de). The Tycho supercomputer hosted at the SCIENCE HPC + center at the University of Copenhagen was used for supporting this + work.

+ + + + + + + + Astropy Collaboration + + Astropy: A community Python package for astronomy + Astronomy and Astrophysics + 201310 + 558 + http://adsabs.harvard.edu/abs/2013A%26A...558A..33A + 10.1051/0004-6361/201322068 + + + + + + Benitez-LlambayAlejandro + + Py-sphviewer: Py-SPHViewer v1.0.0 + Zenodo + 201507 + https://doi.org/10.5281/zenodo.21703 + 10.5281/zenodo.21703 + + + + + + SchallerMatthieu + others + + Swift: A modern highly-parallel gravity and smoothed particle hydrodynamics solver for astrophysical and cosmological applications + arXiv e-prints + 202305 + https://arxiv.org/abs/2305.13380 + 10.48550/arXiv.2305.13380 + arXiv:2305.13380 + + + + + + + TurkM. J. + SmithB. D. + OishiJ. S. + SkoryS. + SkillmanS. W. + AbelT. + NormanM. L. + + yt: A Multi-code Analysis Toolkit for Astrophysical Simulation Data + The Astrophysical Journal Supplement Series + 201101 + 192 + https://arxiv.org/abs/1011.3514 + 10.1088/0067-0049/192/1/9 + 9 + + + + + + + BorrowJosh + BorrisovAlexei + + Swiftsimio: A python library for reading SWIFT data + Journal of Open Source Software + The Open Journal + 2020 + 5 + 52 + https://doi.org/10.21105/joss.02430 + 10.21105/joss.02430 + 2430 + + + + + + + LamSiu Kwan + PitrouAntoine + SeibertStanley + + Numba: A LLVM-based python JIT compiler + Proceedings of the second workshop on the LLVM compiler infrastructure in HPC + Association for Computing Machinery + New York, NY, USA + 2015 + 9781450340052 + https://doi.org/10.1145/2833157.2833162 + 10.1145/2833157.2833162 + + + + + + OkutaRyosuke + UnnoYuya + NishinoDaisuke + HidoShohei + LoomisCrissman + + CuPy: A NumPy-compatible library for NVIDIA GPU calculations + Proceedings of workshop on machine learning systems (LearningSys) in the thirty-first annual conference on neural information processing systems (NIPS) + 2017 + http://learningsys.org/nips17/assets/papers/paper_16.pdf + + + + + + KarrasTero + + Maximizing parallelism in the construction of BVHs, octrees, and k-d trees + Proceedings of the fourth ACM SIGGRAPH / eurographics conference on high-performance graphics + Eurographics Association + Goslar, DEU + 2012 + 9783905674415 + 10.2312/EGGH/HPG12/033-037 + 33 + 37 + + + + + + KellerSebastian + CavelanAurélien + CabezonRubén + MayerLucio + CiorbaFlorina M. + + Cornerstone: Octree Construction Algorithms for Scalable Particle Simulations + arXiv e-prints + 202307 + https://arxiv.org/abs/2307.06345 + 10.48550/arXiv.2307.06345 + arXiv:2307.06345 + + + + + + + SpringelVolker + + E pur si muove: Galilean-invariant cosmological hydrodynamical simulations on a moving mesh + Monthly Notices of the RAS + 201001 + 401 + 2 + https://arxiv.org/abs/0901.4107 + 10.1111/j.1365-2966.2009.15715.x + 791 + 851 + + + + + + van der WaltStéfan + ColbertS. Chris + VaroquauxGaël + + The NumPy Array: A Structure for Efficient Numerical Computation + Computing in Science and Engineering + 201103 + 13 + 2 + https://arxiv.org/abs/1102.1523 + 10.1109/MCSE.2011.37 + 22 + 30 + + + + + + HarrisCharles R. + MillmanK. Jarrod + van der WaltStéfan J. + GommersRalf + VirtanenPauli + CournapeauDavid + WieserEric + TaylorJulian + BergSebastian + SmithNathaniel J. + KernRobert + PicusMatti + HoyerStephan + van KerkwijkMarten H. + BrettMatthew + HaldaneAllan + del Rı́oJaime Fernández + WiebeMark + PetersonPearu + Gérard-MarchantPierre + SheppardKevin + ReddyTyler + WeckesserWarren + AbbasiHameer + GohlkeChristoph + OliphantTravis E. + + Array programming with NumPy + Nature + 202009 + 585 + 7825 + https://arxiv.org/abs/2006.10256 + 10.1038/s41586-020-2649-2 + 357 + 362 + + + + + + ColletteAndrew + + Python and HDF5 + O’Reilly + 2013 + + + + + + PakmorRüdiger + SpringelVolker + + Simulations of magnetic fields in isolated disc galaxies + Monthly Notices of the RAS + 201306 + 432 + 1 + https://arxiv.org/abs/1212.1452 + 10.1093/mnras/stt428 + 176 + 193 + + + + + + SpringelVolker + WhiteSimon D. M. + JenkinsAdrian + FrenkCarlos S. + YoshidaNaoki + GaoLiang + NavarroJulio + ThackerRobert + CrotonDarren + HellyJohn + PeacockJohn A. + ColeShaun + ThomasPeter + CouchmanHugh + EvrardAugust + ColbergJörg + PearceFrazer + + Simulations of the formation, evolution and clustering of galaxies and quasars + Nature + 200506 + 435 + 7042 + https://arxiv.org/abs/astro-ph/0504097 + 10.1038/nature03597 + 629 + 636 + + + + + + WeinbergerRainer + SpringelVolker + PakmorRüdiger + + The AREPO Public Code Release + Astrophysical Journal, Supplement + 202006 + 248 + 2 + https://arxiv.org/abs/1909.04667 + 10.3847/1538-4365/ab908c + 32 + + + + + + + BehnelStefan + BradshawRobert + CitroCraig + DalcinLisandro + SeljebotnDag Sverre + SmithKurt + + Cython: The Best of Both Worlds + Computing in Science and Engineering + 201103 + 13 + 2 + 10.1109/MCSE.2010.118 + 31 + 39 + + + + + + VirtanenPauli + GommersRalf + OliphantTravis E. + HaberlandMatt + ReddyTyler + CournapeauDavid + BurovskiEvgeni + PetersonPearu + WeckesserWarren + BrightJonathan + van der WaltStéfan J. + BrettMatthew + WilsonJoshua + MillmanK. Jarrod + MayorovNikolay + NelsonAndrew R. J. + JonesEric + KernRobert + LarsonEric + CareyC. J. + Polatİlhan + FengYu + MooreEric W. + VanderPlasJake + LaxaldeDenis + PerktoldJosef + CimrmanRobert + HenriksenIan + QuinteroE. A. + HarrisCharles R. + ArchibaldAnne M. + RibeiroAntônio H. + PedregosaFabian + van MulbregtPaul + SciPy 1. 0 Contributors + + SciPy 1.0: fundamental algorithms for scientific computing in Python + Nature Methods + 202002 + 17 + https://arxiv.org/abs/1907.10121 + 10.1038/s41592-019-0686-2 + 261 + 272 + + + + + + KrekelHolger + OliveiraBruno + PfannschmidtRonny + BruynoogheFloris + LaugherBrianna + BruhinFlorian + + Pytest 7.4 + 2004 + https://github.com/pytest-dev/pytest + + + + + + KluyverThomas + Ragan-KelleyBenjamin + PérezFernando + GrangerBrian + BussonnierMatthias + FredericJonathan + KelleyKyle + HamrickJessica + GroutJason + CorlaySylvain + IvanovPaul + AvilaDamián + AbdallaSafia + WillingCarol + teamJupyter development + + Jupyter notebooks – a publishing format for reproducible computational workflows + Positioning and power in academic publishing: Players, agents and agendas + + LoizidesFernando + ScmidtBirgit + + IOS Press + 2016 + https://eprints.soton.ac.uk/403913/ + 10.3233/978-1-61499-649-1-87 + 87 + 90 + + + + + + PerezFernando + GrangerBrian E. + + IPython: A System for Interactive Scientific Computing + Computing in Science and Engineering + 200701 + 9 + 3 + 10.1109/MCSE.2007.53 + 21 + 29 + + + + + + VogelsbergerMark + MarinacciFederico + TorreyPaul + PuchweinEwald + + Cosmological simulations of galaxy formation + Nature Reviews Physics + 202001 + 2 + 1 + https://arxiv.org/abs/1909.07976 + 10.1038/s42254-019-0127-2 + 42 + 66 + + + + + + BerlokThomas + + Hydromagnetic waves in an expanding universe - cosmological MHD code tests using analytic solutions + Monthly Notices of the RAS + 202209 + 515 + 3 + https://arxiv.org/abs/2203.11887 + 10.1093/mnras/stac1882 + 3492 + 3511 + + + + + +

See http://yt-project.org/.

+ + +

See e.g. Pakmor & Springel + (2013) + and Berlok + (2022) + for detailed discussions and a derivation of the comoving + magnetohydrodynamic equations used in cosmological simulations + including magnetic fields.

+ + + +