diff --git a/docs/user/BorealisIO.md b/docs/user/BorealisIO.md index fcfb9f7..30fe9ce 100644 --- a/docs/user/BorealisIO.md +++ b/docs/user/BorealisIO.md @@ -10,42 +10,43 @@ HDF5, see the website of the [HDF Group](www.hdfgroup.org). The Borealis software writes data files in HDF5 format. The files written on site are written record-by-record, in a similar style to the SuperDARN standard -dmap format. These files are named with a .site extension and are said to be -`'site'` structured. +dmap format. As of Borealis v1.0, this is the only structure of HDF5 file supported by Borealis. -After recording, the files are array restructured using pyDARNio, for distribution. -The restructuring is done to make the files more human-readable, and it also -reduces the file size. After restructuring the files are said to be `'array'` -structured. +## Structure of the data -Restructuring reduces repetition by writing file-wide parameters only once, -and writing integration-specific parameters in arrays where the first -dimension is the record index. +Prior to Borealis v1.0, two structures of data file were supported: "site" and "array". The site +structure stores data record-by-record, with some metadata fields redundantly stored in each +record of the file. To reduce file sizes, array-structured files were created. These files +group all like-data from across records into single datasets, zero-padding where necessary to +create numpy arrays. Static metadata fields are stored only once. Site-structured files are denoted +with an additional `.site` suffix to the file extension. As of Borealis v1.0, the conventional file naming +has been changed, so all files end in the `.h5` file extension. This is both for brevity and to distinguish +between the vastly different formats by the file name alone. The restructuring process is fully built into the IO so that if you would like to see the record-by-record data, you can simply return the record's attribute of the IO class for any Borealis file. Similarly, if you would like to see the data in the arrays format, return the arrays attribute. This works regardless of how -the original file was structured. +the original file was structured. For Borealis v1.0 files, I/O is only conducted when +either the `records` or `arrays` attribute is accessed, returning the data in the respective +structure in memory. + +## File types In addition to file structure, there are various types of data sets (file types) that can be produced by Borealis. The file types that can be produced are: - - `'rawrf'` This is the raw samples at the receive bandwidth rate. This is rarely produced and only would be done by request. - - `'antennas_iq'` Downsampled data from individual antennas, i and q samples. - - `'bfiq'` Beamformed i and q samples. Typically two array data sets are included, for main array and interferometer array. - - `'rawacf'` The correlated data given as lags x ranges, for the two arrays. @@ -58,7 +59,7 @@ BorealisRead class takes 3 parameters: - `filename`, - `borealis_filetype`, and -- `borealis_file_structure` (optional but recommended). +- `borealis_file_structure` (optional but recommended for v0.x data, unused for v1.0 onwards). The BorealisRead class can return either array or site structured data, regardless of the file's structure. Note that if you are returning the structure @@ -85,7 +86,7 @@ record_names = borealis_reader.record_names array_data = borealis_reader.arrays ``` -If you don't supply the borealis_file_structure parameter, the reader will +For v0.x data, if you don't supply the borealis_file_structure parameter, the reader will attempt to read the file as array structured first (as this should be the most common structure available to the user), and following failure will attempt to read as site structured. @@ -129,6 +130,9 @@ and site structured files (they vary slightly), see the Borealis documentation ## Writing with BorealisWrite +!!! Warning + Writing Borealis v1.0 data is not supported + The BorealisWrite class takes 4 parameters: - `filename`, @@ -161,3 +165,25 @@ writer = pydarnio.BorealisWrite(my_file, my_rawacf_data, 'rawacf') print(writer.borealis_file_structure) # to check the file structure written ``` + +## Reading data in with xarray + +Borealis v1.0 data can be read in using the [xarray](https://docs.xarray.dev/en/stable/) library. +To do so, you must install pydarnio with the optional `xarray` dependencies. +This can be done using `pip install pydarnio[xarray]`. + +Two flavours of xarray I/O are supported: record-by-record, or with field grouped together across records. + +```python +import pydarnio +infile = "/path/to/data.rawacf.h5" + +# Read in record-by-record (returns list[xarray.Dataset]) +dsets = pydarnio.BorealisV1Read.records_as_xarray(infile) + +# Read in with data grouped together across records (returns xarray.Dataset) +ds = pydarnio.BorealisV1Read.arrays_as_xarray(infile) +``` + +The xarray I/O provides an extremely useful interface for exploring data files, +selecting/slicing data, and plotting data. diff --git a/docs/user/install.md b/docs/user/install.md index 4f122e1..645e143 100644 --- a/docs/user/install.md +++ b/docs/user/install.md @@ -12,7 +12,7 @@ author(s) Marina Schmidt--> ## Prerequisites -**python 3.7+** +**python 3.8+** | Ubuntu | OpenSuse | Fedora | OSX | | ----------- | -------------- | ------------- | ------------- | diff --git a/setup.cfg b/setup.cfg index d5e940f..29ccd83 100644 --- a/setup.cfg +++ b/setup.cfg @@ -21,6 +21,11 @@ install_requires = h5py>=3.11.0 pathlib2 +[options.extras_require] +xarray = + h5netcdf + xarray + [options.packages.find] exclude = test* diff --git a/setup.py b/setup.py deleted file mode 100644 index 785d3c7..0000000 --- a/setup.py +++ /dev/null @@ -1,55 +0,0 @@ -""" -Copyright 2018 SuperDARN Canada, University of Saskatchewan - -setup.py -2018-11-05 -To setup pyDARNio as a third party library. Include installing need libraries for -running the files. - -author: -Marina Schmidt - -Disclaimer: -pyDARNio is under the LGPL v3 license found in the root directory LICENSE.md -Everyone is permitted to copy and distribute verbatim copies of this license -document, but changing it is not allowed. - -This version of the GNU Lesser General Public License incorporates the terms -and conditions of version 3 of the GNU General Public License, -supplemented by the additional permissions listed below. - -""" - -from os import path -from setuptools import setup, find_packages -import sys -from subprocess import check_call -from setuptools.command.install import install, orig - -this_directory = path.abspath(path.dirname(__file__)) -with open(path.join(this_directory, 'README.md'), encoding='utf-8') as f: - long_description = f.read() - - -# Setup information -setup( - name="pydarnio", - version="1.3", - long_description=long_description, - long_description_content_type='text/markdown', - description="Python library for reading and writing SuperDARN data", - url='https://github.com/SuperDARN/pyDARNio.git', - classifiers=[ - 'Development Status :: 5 - Production/Stable', - 'License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)', - 'Programming Language :: Python :: 3.8'], - python_requires='>=3.8', - packages=find_packages(exclude=['docs', 'test']), - author="SuperDARN", - include_package_data=True, - setup_requires=['pyyaml', 'numpy', - 'h5py>=3.11.0', 'pathlib2'], - # pyyaml library install - install_requires=['pyyaml', 'numpy', - 'h5py>=3.11.0', 'pathlib2'] -) diff --git a/pydarnio/__init__.py b/src/pydarnio/__init__.py similarity index 95% rename from pydarnio/__init__.py rename to src/pydarnio/__init__.py index f0cbe1f..eff2d13 100644 --- a/pydarnio/__init__.py +++ b/src/pydarnio/__init__.py @@ -11,8 +11,6 @@ """ # KEEP THIS FILE AS MINIMAL AS POSSIBLE! -import os - # Importing pydarnio exception classes from .exceptions import dmap_exceptions from .exceptions import superdarn_exceptions @@ -44,3 +42,4 @@ from .borealis.borealis import BorealisWrite from .borealis.borealis_convert import BorealisConvert from .borealis.borealis_restructure import BorealisRestructure +from .borealis.v1_onwards import BorealisV1Read, BorealisV1Convert diff --git a/pydarnio/borealis/__init__.py b/src/pydarnio/borealis/__init__.py similarity index 100% rename from pydarnio/borealis/__init__.py rename to src/pydarnio/borealis/__init__.py diff --git a/pydarnio/borealis/base_format.py b/src/pydarnio/borealis/base_format.py similarity index 100% rename from pydarnio/borealis/base_format.py rename to src/pydarnio/borealis/base_format.py diff --git a/pydarnio/borealis/borealis.py b/src/pydarnio/borealis/borealis.py similarity index 75% rename from pydarnio/borealis/borealis.py rename to src/pydarnio/borealis/borealis.py index 331f9ca..e82768d 100644 --- a/pydarnio/borealis/borealis.py +++ b/src/pydarnio/borealis/borealis.py @@ -38,13 +38,14 @@ """ import logging - from collections import OrderedDict from typing import Union +import h5py from pydarnio import borealis_exceptions from .borealis_site import BorealisSiteRead, BorealisSiteWrite from .borealis_array import BorealisArrayRead, BorealisArrayWrite +from .v1_onwards import BorealisV1Read pyDARNio_log = logging.getLogger('pyDARNio') @@ -129,21 +130,35 @@ def __init__(self, filename: str, borealis_filetype: str, self.filename, borealis_filetype) self.borealis_filetype = borealis_filetype - if borealis_file_structure is None: - self._reader, self._borealis_file_structure = \ - self.return_reader(self.filename, self.borealis_filetype) - elif borealis_file_structure == 'site': - self._reader = BorealisSiteRead(self.filename, - self.borealis_filetype) - self._borealis_file_structure = 'site' - elif borealis_file_structure == 'array': - self._reader = BorealisArrayRead(self.filename, - self.borealis_filetype) - self._borealis_file_structure = 'array' - else: # unknown structure - raise borealis_exceptions.\ - BorealisStructureError("Unknown structure type: {}" - "".format(borealis_file_structure)) + with h5py.File(self.filename, 'r') as f: + keys = sorted(list(f.keys())) + if 'borealis_git_hash' in f.attrs.keys(): + version = f.attrs['borealis_git_hash'] + elif 'borealis_git_hash' in f[keys[0]].attrs.keys(): + version = f[keys[0]].attrs['borealis_git_hash'] + elif 'borealis_git_hash' in f[keys[0]].keys(): + version = f[keys[0]]['borealis_git_hash'][()][:].decode('utf-8') + else: + raise borealis_exceptions.BorealisVersionError("Unable to determine Borealis version") + self.version = [int(x) for x in version.lstrip('v').split('-')[0].split('.')] + # version is 'v0.6.1-abcdefg' or similar + + if self.version[0] < 1: + if borealis_file_structure is None: + self._reader, self._borealis_file_structure = \ + self.return_reader(self.filename, self.borealis_filetype) + elif borealis_file_structure == 'site': + self._reader = BorealisSiteRead(self.filename, + self.borealis_filetype) + self._borealis_file_structure = 'site' + elif borealis_file_structure == 'array': + self._reader = BorealisArrayRead(self.filename, + self.borealis_filetype) + self._borealis_file_structure = 'array' + else: # unknown structure + raise borealis_exceptions.\ + BorealisStructureError("Unknown structure type: {}" + "".format(borealis_file_structure)) def __repr__(self): """ for representation of the class object""" @@ -175,7 +190,10 @@ def record_names(self): These correspond to Borealis file record write times (in ms since epoch), and are equal to the group names in the site file types. """ - return self._reader.record_names + if self.version[0] < 1: + return self._reader.record_names + else: + return [] @property def records(self): @@ -183,7 +201,10 @@ def records(self): The Borealis data in a dictionary of records, according to the site file format. """ - return self._reader.records + if self.version[0] < 1: + return self._reader.records + else: + return BorealisV1Read.read_records(self.filename) @property def arrays(self): @@ -191,7 +212,10 @@ def arrays(self): The Borealis data in a dictionary of arrays, according to the restructured array file format. """ - return self._reader.arrays + if self.version[0] < 1: + return self._reader.arrays + else: + return BorealisV1Read.read_arrays(self.filename) @property def software_version(self): @@ -202,14 +226,20 @@ def software_version(self): May impact the fields included in the file as each version has a different field structure/format """ - return self._reader.software_version + if self.version[0] < 1: + return self._reader.software_version + else: + return f"v{'.'.join(self.version)}" @property def format(self): """ The format class used for the file, from the borealis_formats module. """ - return self._reader.format + if self.version[0] < 1: + return self._reader.format + else: + return None @staticmethod def return_reader(borealis_hdf5_file: str, borealis_filetype: str) -> \ @@ -346,46 +376,54 @@ def __init__(self, filename: str, borealis_data: Union[dict, OrderedDict], self.data = borealis_data self.borealis_filetype = borealis_filetype - if borealis_file_structure is None: - self._writer, self._borealis_file_structure = \ - self.return_writer(self.filename, self.data, - self.borealis_filetype, **kwargs) - elif borealis_file_structure == 'site': - self._writer = BorealisSiteWrite(self.filename, self.data, - self.borealis_filetype, - **kwargs) - self._borealis_file_structure = 'site' - elif borealis_file_structure == 'array': - self._writer = BorealisArrayWrite(self.filename, self.data, - self.borealis_filetype, **kwargs) - self._borealis_file_structure = 'array' - else: # unknown structure - raise borealis_exceptions.\ - BorealisStructureError('Unknown structure type: {}' - ''.format(borealis_file_structure)) + if 'borealis_git_hash' in borealis_data.keys(): + version = borealis_data['borealis_git_hash'] + else: + version = borealis_data[0]['borealis_git_hash'] + self.version = [int(x) for x in version.lstrip('v').split('-')[0].split('.')] + + if self.version[0] < 1: + if borealis_file_structure is None: + self._writer, self._borealis_file_structure = \ + self.return_writer(self.filename, self.data, + self.borealis_filetype, **kwargs) + elif borealis_file_structure == 'site': + self._writer = BorealisSiteWrite(self.filename, self.data, + self.borealis_filetype, + **kwargs) + self._borealis_file_structure = 'site' + elif borealis_file_structure == 'array': + self._writer = BorealisArrayWrite(self.filename, self.data, + self.borealis_filetype, **kwargs) + self._borealis_file_structure = 'array' + else: # unknown structure + raise borealis_exceptions.\ + BorealisStructureError('Unknown structure type: {}' + ''.format(borealis_file_structure)) + else: + raise NotImplementedError("Writing v1.0+ Borealis files is not supported") def __repr__(self): """For representation of the class object""" - return "{class_name}({filename}, {current_record_name})"\ + return "{class_name}({filename})"\ "".format(class_name=self.__class__.__name__, - filename=self.filename, - current_record_name=self.current_record_name) + filename=self.filename) def __str__(self): """For printing of the class object""" - return "Writing to filename: {filename} at record name: "\ - "{current_record_name}"\ - "".format(filename=self.filename, - current_record_name=self.current_record_name) + return "Writing to filename: {filename} ".format(filename=self.filename) @property def borealis_file_structure(self): """ The structure of the file read, 'site' or 'array'. Default None. """ - return self._borealis_file_structure + if self.version[0] < 1: + return self._borealis_file_structure + else: + return None @property def compression(self): @@ -398,7 +436,10 @@ def compression(self): will have 'zlib' compression to enable a faster read time for downstream users. """ - return self._writer.compression + if self.version[0] < 1: + return self._writer.compression + else: + return None @property def record_names(self): @@ -407,7 +448,10 @@ def record_names(self): These correspond to Borealis file record write times (in ms since epoch), and are equal to the group names in the site file types. """ - return self._writer.record_names + if self.version[0] < 1: + return self._writer.record_names + else: + return None @property def records(self): @@ -415,7 +459,10 @@ def records(self): The Borealis data in a dictionary of records, according to the site file format. """ - return self._writer.records + if self.version[0] < 1: + return self._writer.records + else: + return None @property def arrays(self): @@ -423,7 +470,10 @@ def arrays(self): The Borealis data in a dictionary of arrays, according to the restructured array file format. """ - return self._writer.arrays + if self.version[0] < 1: + return self._writer.arrays + else: + return None @property def software_version(self): @@ -434,14 +484,20 @@ def software_version(self): May impact the fields included in the file as each version has a different field structure/format """ - return self._writer.software_version + if self.version[0] < 1: + return self._writer.software_version + else: + return f"v{'.'.join(self.version)}" @property def format(self): """ The format class used for the file, from the borealis_formats module. """ - return self._writer.format + if self.version[0] < 1: + return self._writer.format + else: + return None @staticmethod def return_writer(filename: str, data: Union[dict, OrderedDict], diff --git a/pydarnio/borealis/borealis_array.py b/src/pydarnio/borealis/borealis_array.py similarity index 100% rename from pydarnio/borealis/borealis_array.py rename to src/pydarnio/borealis/borealis_array.py diff --git a/pydarnio/borealis/borealis_convert.py b/src/pydarnio/borealis/borealis_convert.py similarity index 96% rename from pydarnio/borealis/borealis_convert.py rename to src/pydarnio/borealis/borealis_convert.py index 77fae9a..00d31c6 100644 --- a/pydarnio/borealis/borealis_convert.py +++ b/src/pydarnio/borealis/borealis_convert.py @@ -47,55 +47,11 @@ from typing import Union from pydarnio import (borealis_exceptions, BorealisRead, SDarnWrite, dict2dmap) +from pydarnio.borealis.borealis_utilities import code_to_stid +from pydarnio.borealis.v1_onwards import BorealisV1Convert pyDARNio_log = logging.getLogger('pyDARNio') -# 3 letter radar code, mapped to station id for SDarn files conversion. -# TODO: when merged with plotting, remove this dictionary and call the -# one in the plotting folder... also move Radars.py to a more -# central location. -code_to_stid = { - "tst": 0, - "gbr": 1, - "sch": 2, - "kap": 3, - "hal": 4, - "sas": 5, - "pgr": 6, - "kod": 7, - "sto": 8, - "pyk": 9, - "han": 10, - "san": 11, - "sys": 12, - "sye": 13, - "tig": 14, - "ker": 15, - "ksr": 16, - "unw": 18, - "zho": 19, - "mcm": 20, - "fir": 21, - "sps": 22, - "bpk": 24, - "wal": 32, - "bks": 33, - "hok": 40, - "hkw": 41, - "inv": 64, - "rkn": 65, - "cly": 66, - "dce": 96, - "svb": 128, - "fhw": 204, - "fhe": 205, - "cvw": 206, - "cve": 207, - "adw": 208, - "ade": 209, - "ekb": 512, -} - class BorealisConvert(BorealisRead): """ @@ -212,17 +168,18 @@ def __init__(self, borealis_filename: str, borealis_filetype: str, self.sdarn_filename = sdarn_filename self.borealis_filename = self.filename - try: - first_key = list(self.records.keys())[0] - self._borealis_slice_id = self.records[first_key]['slice_id'] - except KeyError as kerr: - if borealis_slice_id is not None: - self._borealis_slice_id = int(borealis_slice_id) - else: - raise borealis_exceptions.BorealisStructureError( - 'The slice_id could not be found in the file: Borealis ' - 'files produced before Borealis v0.5 must provide the ' - 'slice_id value to the BorealisConvert class.') from kerr + if self.version[0] < 1: + try: + first_key = list(self.records.keys())[0] + self._borealis_slice_id = self.records[first_key]['slice_id'] + except KeyError as kerr: + if borealis_slice_id is not None: + self._borealis_slice_id = int(borealis_slice_id) + else: + raise borealis_exceptions.BorealisStructureError( + 'The slice_id could not be found in the file: Borealis ' + 'files produced before Borealis v0.5 must provide the ' + 'slice_id value to the BorealisConvert class.') from kerr self._sdarn_dmap_records = {} self._sdarn_dict = {} @@ -325,11 +282,17 @@ def _convert_records_to_dmap(self): BorealisConversionTypesError """ if self.sdarn_filetype == 'iqdat': - if self._is_convertible_to_iqdat(): - self._convert_bfiq_to_iqdat() + if self.version[0] < 1: + if self._is_convertible_to_iqdat(): + self._convert_bfiq_to_iqdat() + else: + self._sdarn_dmap_records = BorealisV1Convert.bfiq_to_dmap(self.borealis_filename) elif self.sdarn_filetype == 'rawacf': - if self._is_convertible_to_rawacf(): - self._convert_rawacf_to_rawacf() + if self.version[0] < 1: + if self._is_convertible_to_rawacf(): + self._convert_rawacf_to_rawacf() + else: + self._sdarn_dmap_records = BorealisV1Convert.rawacf_to_dmap(self.borealis_filename) else: # nothing else is currently supported raise borealis_exceptions.BorealisConversionTypesError( self.sdarn_filename, self.borealis_filetype, diff --git a/pydarnio/borealis/borealis_formats.py b/src/pydarnio/borealis/borealis_formats.py similarity index 100% rename from pydarnio/borealis/borealis_formats.py rename to src/pydarnio/borealis/borealis_formats.py diff --git a/pydarnio/borealis/borealis_restructure.py b/src/pydarnio/borealis/borealis_restructure.py similarity index 100% rename from pydarnio/borealis/borealis_restructure.py rename to src/pydarnio/borealis/borealis_restructure.py diff --git a/pydarnio/borealis/borealis_site.py b/src/pydarnio/borealis/borealis_site.py similarity index 100% rename from pydarnio/borealis/borealis_site.py rename to src/pydarnio/borealis/borealis_site.py diff --git a/pydarnio/borealis/borealis_utilities.py b/src/pydarnio/borealis/borealis_utilities.py similarity index 97% rename from pydarnio/borealis/borealis_utilities.py rename to src/pydarnio/borealis/borealis_utilities.py index e00fefe..244f5e9 100644 --- a/pydarnio/borealis/borealis_utilities.py +++ b/src/pydarnio/borealis/borealis_utilities.py @@ -637,4 +637,48 @@ def pulse_phase_offset_site_fix(data_dict: dict): continue ppo = rec['pulse_phase_offset'] if ppo.shape != rec['pulses'].shape: # The field is broken, was empty when written to file and so was restructured improperly - rec['pulse_phase_offset'] = np.zeros(rec['pulses'].shape, dtype=np.float32) \ No newline at end of file + rec['pulse_phase_offset'] = np.zeros(rec['pulses'].shape, dtype=np.float32) + + +# 3 letter radar code, mapped to station id for SDarn files conversion. +code_to_stid = { + "tst": 0, + "gbr": 1, + "sch": 2, + "kap": 3, + "hal": 4, + "sas": 5, + "pgr": 6, + "kod": 7, + "sto": 8, + "pyk": 9, + "han": 10, + "san": 11, + "sys": 12, + "sye": 13, + "tig": 14, + "ker": 15, + "ksr": 16, + "unw": 18, + "zho": 19, + "mcm": 20, + "fir": 21, + "sps": 22, + "bpk": 24, + "wal": 32, + "bks": 33, + "hok": 40, + "hkw": 41, + "inv": 64, + "rkn": 65, + "cly": 66, + "dce": 96, + "svb": 128, + "fhw": 204, + "fhe": 205, + "cvw": 206, + "cve": 207, + "adw": 208, + "ade": 209, + "ekb": 512, +} diff --git a/src/pydarnio/borealis/v1_onwards.py b/src/pydarnio/borealis/v1_onwards.py new file mode 100644 index 0000000..f2d69e5 --- /dev/null +++ b/src/pydarnio/borealis/v1_onwards.py @@ -0,0 +1,634 @@ +# Copyright 2019 SuperDARN Canada, University of Saskatchewan +# Author: Marci Detwiller +""" +This file contains classes and functions for +converting of Borealis v1.0+ file types. + +Classes +------- +BorealisRead +BorealisWrite + +Exceptions +---------- +BorealisFileTypeError +BorealisFieldMissingError +BorealisExtraFieldError +BorealisDataFormatTypeError +BorealisNumberOfRecordsError +BorealisConversionTypesError +BorealisConvert2IqdatError +BorealisConvert2RawacfError +ConvertFileOverWriteError + +See Also +-------- +BorealisConvert + +For more information on Borealis data files and how they convert to SDARN +filetypes, see: https://borealis.readthedocs.io/en/latest/ +""" + +import logging +from datetime import datetime + +import h5py +import numpy as np + +from pydarnio import borealis_exceptions +from pydarnio.borealis.borealis_utilities import code_to_stid + +pyDARNio_log = logging.getLogger('pyDARNio') + + +class BorealisV1Read: + """ + Class for reading Borealis v1.0+ filetypes. + + See Also + -------- + BaseFormat + BorealisRawacf + BorealisBfiq + BorealisAntennasIq + BorealisRawrf + """ + + @staticmethod + def read_records(filename: str): + """ + Reads in records and metadata from the file. + """ + records = list() + metadata = dict() + with h5py.File(filename, 'r') as f: + metadata_group = f['metadata'] + for k in list(metadata_group.keys()): + metadata[k] = metadata_group[k][()] + rec_names = sorted(list(f.keys())) + rec_names.remove('metadata') + for name in rec_names: + rec = f[name] + rec_dict = dict() + for k in list(rec.keys()): + if k not in metadata.keys(): + rec_dict[k] = rec[k][()] + records.append(rec_dict) + + return records, metadata + + @staticmethod + def records_as_xarray(filename: str): + try: + import xarray as xr + datasets = list() + with h5py.File(filename, 'r') as f: + keys = sorted(list(f.keys())) + keys.remove("metadata") + for key in keys: + ds = xr.open_dataset(filename, group=f"/{key}", phony_dims="access") + datasets.append(ds) + return datasets + except ImportError: + raise ImportError("Unable to import xarray. Ensure that you have xarray installed with the `h5netcdf` " + "engine (this can be installed by installing pydarnio with the `xarray` option, e.g. " + "`pip install pydarnio[xarray]`)") + + @staticmethod + def read_arrays(filename: str): + """ + Reads in records and metadata from the file, combining like-fields of records into + single large arrays. + """ + aveperiod_indices = [] + beam_indices = [] + bfiq_indices = [] + metadata = dict() + with h5py.File(filename, 'r') as f: + metadata_group = f['metadata'] + for k in list(metadata_group.keys()): + metadata[k] = metadata_group[k][()] + rec_names = sorted(list(f.keys())) + rec_names.remove('metadata') + field_names = sorted(list(f[rec_names[0]].keys())) + for metadata_name in metadata.keys(): + field_names.remove(metadata_name) + + fields_lists = {k: list() for k in field_names} + for i, name in enumerate(rec_names): + rec = f[name] + for k in field_names: + data = rec[k][()] + if k == 'sqn_timestamps': + aveperiod_indices.extend([i] * len(data)) + if k == 'beam_nums': + beam_indices.extend([i] * len(data)) + if k == 'bfiq_data': + bfiq_indices.extend([i] * data.shape[1] * data.shape[2]) + data = data.reshape((data.shape[0], -1, data.shape[3])) + if k in ['beam_nums', 'beam_azms', 'rx_main_excitations', 'rx_intf_excitations', 'tx_excitations']: + data = data.flatten() + if k in ['intf_acfs', 'main_acfs', 'xcfs']: + data = data.reshape((-1,) + data.shape[1:]) + fields_lists[k].append(data) + + rec_dict = dict() + for k, v in fields_lists.items(): + if k in ['antennas_iq_data', 'bfiq_data', 'rawrf_data']: + rec_dict[k] = np.concatenate(v, axis=1) + elif k in ['beam_nums', 'beam_azms', 'pulse_phase_offset', 'sqn_timestamps', + 'intf_acfs', 'main_acfs', 'xcfs']: + rec_dict[k] = np.concatenate(v, axis=0) + else: + rec_dict[k] = np.stack(v, axis=0) + rec_dict.update(metadata) + rec_dict['aveperiod_indices'] = np.array(aveperiod_indices, dtype=np.uint32) + rec_dict['aveperiod'] = np.arange(len(rec_names), dtype=np.uint32) + rec_dict['beam_indices'] = np.array(beam_indices, dtype=np.uint32) + if 'bfiq_data' in rec_dict.keys(): + rec_dict['bfiq_indices'] = np.array(bfiq_indices, dtype=np.uint32) + + return rec_dict + + @staticmethod + def arrays_as_xarray(filename: str): + """ + Reads in the file as array-formatted fields, using xarray. + """ + try: + import xarray as xr + except ImportError: + raise ImportError("Unable to import xarray. Ensure that you have xarray installed with the `h5netcdf` " + "engine (this can be installed by installing pydarnio with the `xarray` option, e.g. " + "`pip install pydarnio[xarray]`)") + + arrays = BorealisV1Read.read_arrays(filename) + + coord_fields = { + "aveperiod": ("aveperiod", ""), + "aveperiod_indices": ("aveperiod_idx", "index into `aveperiod`"), + "beam_indices": ("beam", "index into `aveperiod`"), + "bfiq_indices": ("sequence_times_beam", "index into `aveperiod`"), + "blanked_samples": ("blanks", ""), + "beam_nums": ("beam_num", ""), + "coordinates": ("coord", ""), + "antenna_arrays": ("array", ""), + "antennas": ("antenna", ""), + "cfs_freqs": ("freq", "Hz"), + "lags": ("lag", ""), + "lag_numbers": ("lag number", "`tau_spacing`"), + "lag_pulse_descriptors": ("pulse_descriptor", ""), + "pulses": ("pulse", "`tau_spacing`"), + "range_gates": ("range_gate", ""), + "rx_antennas": ("rx_antenna", ""), + "rx_main_antennas": ("main_antenna", ""), + "rx_intf_antennas": ("intf_antenna", ""), + "sample_time": ("sample_time", "μs"), + "sqn_timestamps": ("sequence", ""), + "tx_antennas": ("tx_antenna", ""), + } + data_fields = { + "agc_status_word": ["aveperiod"], + "antenna_locations": ["antennas", "coordinates"], + "antennas_iq_data": ["rx_antennas", "sqn_timestamps", "sample_time"], + "averaging_method": [], + "beam_azms": ["beam_nums"], + "bfiq_data": ["antenna_arrays", "bfiq_indices", "sample_time"], + "borealis_git_hash": [], + "cfs_masks": ["aveperiod", "cfs_freqs"], + "cfs_noise": ["aveperiod", "cfs_freqs"], + "cfs_range": [], + "data_normalization_factor": [], + "experiment_comment": [], + "experiment_id": [], + "experiment_name": [], + "first_range": [], + "first_range_rtt": [], + "freq": ["aveperiod"], + "gps_locked": ["aveperiod"], + "gps_to_system_time_diff": ["aveperiod"], + "int_time": ["aveperiod"], + "intf_acfs": ["beam_nums", "range_gates", "lag_numbers"], + "lag_pulses": ["lags", "lag_pulse_descriptors"], + "lp_status_word": ["aveperiod"], + "main_acfs": ["beam_nums", "range_gates", "lag_numbers"], + "num_sequences": ["aveperiod"], + "num_slices": [], + "pulse_phase_offset": ["sqn_timestamps", "pulses"], + "range_sep": [], + "rawrf_data": ["rx_antennas", "sqn_timestamps", "sample_time"], + "rx_center_freq": [], + "rx_sample_rate": [], + "rx_main_excitations": ["beam_nums", "rx_main_antennas"], + "rx_intf_excitations": ["beam_nums", "rx_intf_antennas"], + "samples_data_type": [], + "scan_start_marker": ["aveperiod"], + "scheduling_mode": [], + "slice_comment": [], + "slice_id": [], + "slice_interfacing": [], + "station": [], + "station_location": ["coordinates"], + "tau_spacing": [], + "tx_excitations": ["aveperiod", "tx_antennas"], + "tx_pulse_len": [], + "xcfs": ["beam_nums", "range_gates", "lag_numbers"] + } + + data_arrays = dict() + for k, dims in data_fields.items(): + if k not in arrays.keys(): + continue + coords = list() + for d in dims: + data = arrays[d] + if d == 'sqn_timestamps': + data = np.array([datetime.utcfromtimestamp(x) for x in data]) + coords.append((coord_fields[d][0], data, {"units": coord_fields[d][1]})) + if len(coords) == 0: + coords = None + data_arrays[k] = xr.DataArray(arrays[k], coords) + + return xr.Dataset(data_arrays) + + +class BorealisV1Convert: + @staticmethod + def bfiq_to_dmap(filename: str): + """ + Conversion for bfiq to iqdat SDARN DMap records. + + See Also + -------- + __convert_bfiq_record + https://superdarn.github.io/rst/superdarn/src.doc/rfc/0027.html + https://borealis.readthedocs.io/en/master/ + BorealisBfiq + Iqdat + + Raises + ------ + BorealisConvert2IqdatError + + Notes + ----- + SuperDARN RFC 0027 specifies that the dimensions of the data in + iqdat should be by number of sequences, number of arrays, number + of samples, 2 (i+q). There is some history where the dimensions were + instead sequences, samples, arrays, 2(i+q). We have chosen to + use the former, as it is consistent with the rest of SuperDARN Canada + radars at this time and is as specified in the document. This means + that you may need to use make_raw with the -d option in RST if you + wish to process the resulting iqdat into rawacf. + + Returns + ------- + dmap_recs, the records converted to DMap format + """ + try: + recs = [] + records, metadata = BorealisV1Read.read_records(filename) + for record in records: + record_dict_list = BorealisV1Convert.convert_bfiq_record( + record, + metadata, + filename, + ) + recs.extend(record_dict_list) + except Exception as e: + raise borealis_exceptions.BorealisConvert2IqdatError(e) from e + return recs + + @staticmethod + def convert_bfiq_record(bfiq_dict: dict, metadata_dict: dict, origin_string: str) -> list: + """ + Converts a single record dict of Borealis bfiq data to a SDARN DMap + record dict. + + Parameters + ---------- + bfiq_dict: dict + dictionary of bfiq record data. + metadata_dict: dict + dictionary of file-level metadata. + origin_string: str + String representing origin of the Borealis data, typically + Borealis filename. + """ + record_dict = {**bfiq_dict, **metadata_dict} + # data_descriptors (dimensions) are num_antenna_arrays, + # num_sequences, num_beams, num_samps + # scale by normalization and then scale to integer max as per + # dmap style + data = ( + record_dict['bfiq_data'] / + record_dict['data_normalization_factor'] * + np.iinfo(np.int16).max + ) + + githash = record_dict['borealis_git_hash'][:].decode('utf-8') + version = githash.lstrip('v').split('-')[0].split('.') + borealis_major_revision = int(version[0]) + borealis_minor_revision = int(version[1]) + + # base offset for setting the toff field in SDARN DMap iqdat file. + offset = 2 * record_dict['antenna_arrays'].shape[0] * data.shape[-1] + + record_dict_list = [] + for beam_index, beam in enumerate(record_dict['beam_nums']): + # grab this beam's data + # shape is now num_antenna_arrays x num_sequences + # x num_samps + this_data = data[:, :, beam_index, :] + # iqdat shape is num_sequences x num_antennas_arrays x + # num_samps x 2 (real, imag), flattened + reshaped_data = [] + for i in range(record_dict['num_sequences']): + # get the samples for each array 1 after the other + arrays = [this_data[x, i, :] + for x in range(this_data.shape[0])] + # append + reshaped_data.append(np.ravel(arrays)) + + # (num_sequences x num_antenna_arrays x num_samps, + # flattened) + flattened_data = np.array(reshaped_data).flatten() + + int_data = np.empty(flattened_data.size * 2, dtype=np.float64) + int_data[0::2] = flattened_data.real + int_data[1::2] = flattened_data.imag + + np.minimum(int_data, 32767, int_data) + np.maximum(int_data, -32768, int_data) + + int_data = np.array(int_data, dtype=np.int16) + + start_time = datetime.utcfromtimestamp(record_dict['sqn_timestamps'][0]) + + # flattening done in convert_to_dmap_datastructures + sdarn_record_dict = { + 'radar.revision.major': np.int8(borealis_major_revision), + 'radar.revision.minor': np.int8(borealis_minor_revision), + 'origin.code': np.int8(100), # indicating Borealis + 'origin.time': datetime.utcfromtimestamp(record_dict['sqn_timestamps'][0]).strftime("%c"), + 'origin.command': 'Borealis ' + githash + ' ' + record_dict['experiment_name'][:].decode('utf-8'), + 'cp': np.int16(record_dict['experiment_id']), + 'stid': np.int16(code_to_stid[record_dict['station'][:].decode('utf-8')]), + 'time.yr': np.int16(start_time.year), + 'time.mo': np.int16(start_time.month), + 'time.dy': np.int16(start_time.day), + 'time.hr': np.int16(start_time.hour), + 'time.mt': np.int16(start_time.minute), + 'time.sc': np.int16(start_time.second), + 'time.us': np.int32(start_time.microsecond), + 'txpow': np.int16(-1), + 'nave': np.int16(record_dict['num_sequences']), + 'atten': np.int16(0), + 'lagfr': np.int16(record_dict['first_range_rtt']), + # smsep is in us; conversion from seconds + 'smsep': np.int16(1e6 / record_dict['rx_sample_rate']), + 'ercod': np.int16(0), + 'stat.agc': np.int16(record_dict['agc_status_word']), + 'stat.lopwr': np.int16(record_dict['lp_status_word']), + # TODO: currently not implemented + 'noise.search': np.float32(0.0), + # TODO: currently not implemented + 'noise.mean': np.float32(0), + 'channel': np.int16(record_dict['slice_id']), + 'bmnum': np.int16(beam), + 'bmazm': np.float32(record_dict['beam_azms'][beam_index]), + 'scan': np.int16(record_dict['scan_start_marker']), + # no digital receiver offset or rxrise required in + # Borealis + 'offset': np.int16(0), + 'rxrise': np.int16(0), + 'intt.sc': np.int16(np.floor(record_dict['int_time'])), + 'intt.us': np.int32(np.fmod(record_dict['int_time'], 1.0) * 1e6), + 'txpl': np.int16(record_dict['tx_pulse_len']), + 'mpinc': np.int16(record_dict['tau_spacing']), + 'mppul': np.int16(len(record_dict['pulses'])), + # an alternate lag-zero will be given, so subtract 1. + 'mplgs': np.int16(record_dict['lag_numbers'].shape[0] - 1), + 'nrang': np.int16(len(record_dict['range_gates'])), + 'frang': np.int16(round(record_dict['first_range'])), + 'rsep': np.int16(round(record_dict['range_sep'])), + 'xcf': np.int16('intf' in record_dict['antenna_arrays']), + 'tfreq': np.int16(record_dict['freq']), + # mxpwr filler; cannot specify this information + 'mxpwr': np.int32(-1), + # lvmax RST default + 'lvmax': np.int32(20000), + 'iqdata.revision.major': np.int32(1), + 'iqdata.revision.minor': np.int32(0), + 'combf': 'Converted from Borealis file: ' + origin_string + + ' ; Number of beams in record: ' + str(len(record_dict['beam_nums'])) + + ' ; ' + record_dict['experiment_comment'][:].decode('utf-8') + + ' ; ' + record_dict['slice_comment'][:].decode('utf-8'), + 'seqnum': np.int32(record_dict['num_sequences']), + 'chnnum': np.int32(record_dict['antenna_arrays'].shape[0]), + 'smpnum': np.int32(len(record_dict['sample_time'])), + # NOTE: The following is a hack. This is currently how + # iqdat files are being processed . RST make_raw does + # not use first range information at all, only skip + # number. + # However ROS provides the number of ranges to the + # first range as the skip number. Skip number is + # documented as number to identify bad ranges due + # to digital receiver rise time. Borealis skpnum should + # in theory =0 as the first sample from Borealis + # decimated (prebfiq) data is centred on the first + # pulse. + 'skpnum': np.int32(record_dict['first_range'] / record_dict['range_sep']), + 'ptab': record_dict['pulses'].astype(np.int16), + 'ltab': record_dict['lag_pulses'].astype(np.int16), + # timestamps in ms, convert to seconds and us. + 'tsc': np.array([np.floor(x/1e3) for x in record_dict['sqn_timestamps']], dtype=np.int32), + 'tus': np.array([np.fmod(x, 1000.0) * 1e3 for x in record_dict['sqn_timestamps']], dtype=np.int32), + 'tatten': np.array([0] * record_dict['num_sequences'], dtype=np.int16), + 'tnoise': np.zeros(record_dict['num_sequences'], dtype=np.float32), + 'toff': np.array([i * offset for i in range(record_dict['num_sequences'])], dtype=np.int32), + 'tsze': np.array([offset] * record_dict['num_sequences'], dtype=np.int32), + 'data': int_data + } + record_dict_list.append(sdarn_record_dict) + return record_dict_list + + @staticmethod + def rawacf_to_dmap(filename: str): + """ + Conversion for Borealis hdf5 rawacf to SDARN DMap rawacf files. + + See Also + -------- + __convert_rawacf_record + https://superdarn.github.io/rst/superdarn/src.doc/rfc/0008.html + https://borealis.readthedocs.io/en/master/ + BorealisRawacf + Rawacf + + Raises + ------ + BorealisConvert2RawacfError + + Returns + ------- + dmap_recs, the records converted to DMap format + """ + try: + recs = [] + records, metadata = BorealisV1Read.read_records(filename) + for record in records: + record_dict_list = BorealisV1Convert.convert_rawacf_record( + record, + metadata, + filename, + ) + recs.extend(record_dict_list) + except Exception as e: + raise borealis_exceptions.BorealisConvert2RawacfError(e) from e + return recs + + @staticmethod + def convert_rawacf_record(rawacf_dict: dict, metadata_dict: dict, origin_string: str) -> list: + """ + Converts a single record dict of Borealis rawacf data to a SDARN DMap + record dict. + + Parameters + ---------- + rawacf_dict : dict + dictionary of rawacf record data. + metadata_dict: dict + dictionary of file-level metadata. + origin_string : str + String representing origin of the Borealis data, typically + Borealis filename. + """ + record_dict = {**rawacf_dict, **metadata_dict} + shaped_data = {} + data_dimensions = record_dict['main_acfs'].shape + + # correlation_descriptors are num_beams, num_ranges, num_lags + # scale by the scale squared to make up for the multiply + # in correlation (integer max squared) + shaped_data['main_acfs'] = record_dict['main_acfs'] * (np.iinfo(np.int16).max**2 / record_dict['data_normalization_factor']**2) + + if 'intf_acfs' in record_dict.keys(): + shaped_data['intf_acfs'] = record_dict['intf_acfs'] * (np.iinfo(np.int16).max**2 / record_dict['data_normalization_factor']**2) + if 'xcfs' in record_dict.keys(): + shaped_data['xcfs'] = record_dict['xcfs'] * (np.iinfo(np.int16).max**2 / record_dict['data_normalization_factor']**2) + + githash = record_dict['borealis_git_hash'][:].decode('utf-8') + version = githash.lstrip('v').split('-')[0].split('.') + borealis_major_revision = int(version[0]) + borealis_minor_revision = int(version[1]) + + record_dict_list = [] + for beam_index, beam in enumerate(record_dict['beam_nums']): + # this beam, all ranges lag 0 + lag_zero = shaped_data['main_acfs'][beam_index, :, 0] + lag_zero_power = np.abs(lag_zero) + + correlation_dict = {} + for key in shaped_data: + # num_ranges x num_lags (complex) + this_correlation = shaped_data[key][beam_index, :, :-1] + + # (num_ranges x num_lags, flattened) + flattened_data = np.array(this_correlation).flatten() + + int_data = np.empty(flattened_data.size * 2, dtype=np.float32) + int_data[0::2] = flattened_data.real + int_data[1::2] = flattened_data.imag + # num_ranges x num_lags x 2; num_lags is one less than + # in Borealis file because Borealis keeps alternate + # lag0 + new_data = int_data.reshape( + data_dimensions[1], + data_dimensions[2]-1, + 2) + # NOTE: Flattening happening in + # convert_to_dmap_datastructures + # place the SDARN-style array in the dict + correlation_dict[key] = new_data + + # TX Antenna Mag only introduced in Borealis v0.7 onwards, so txpow defaults to -1 if not present. + # If present, txpow is a bitfield mapping of whether each antenna was transmitting. Antenna 15 is the + # MSB, and Antenna 0 the LSB. Since txpow is a signed int in DMAP, -1 means all antennas transmitting. + txpow = np.uint16() + if 'tx_excitations' not in record_dict.keys(): + raise ValueError(f'"tx_excitations" not in record: {record_dict.keys()}') + for i in range(len(record_dict['tx_excitations'])): + if np.abs(record_dict['tx_excitations'][i]) > 0: + txpow += 1 << i + + start_time = datetime.utcfromtimestamp(record_dict['sqn_timestamps'][0]) + sdarn_record_dict = { + 'radar.revision.major': np.int8(borealis_major_revision), + 'radar.revision.minor': np.int8(borealis_minor_revision), + 'origin.code': np.int8(100), # indicating Borealis + 'origin.time': datetime.utcfromtimestamp(record_dict['sqn_timestamps'][0]).strftime("%c"), + 'origin.command': 'Borealis ' + githash + ' ' + record_dict['experiment_name'][:].decode('utf-8'), + 'cp': np.int16(record_dict['experiment_id']), + 'stid': np.int16(code_to_stid[record_dict['station'][:].decode('utf-8')]), + 'time.yr': np.int16(start_time.year), + 'time.mo': np.int16(start_time.month), + 'time.dy': np.int16(start_time.day), + 'time.hr': np.int16(start_time.hour), + 'time.mt': np.int16(start_time.minute), + 'time.sc': np.int16(start_time.second), + 'time.us': np.int32(start_time.microsecond), + 'txpow': np.int16(txpow), + # see Borealis documentation + 'nave': np.int16(record_dict['num_sequences']), + 'atten': np.int16(0), + 'lagfr': np.int16(record_dict['first_range_rtt']), + 'smsep': np.int16(1e6/record_dict['rx_sample_rate']), + 'ercod': np.int16(0), + 'stat.agc': np.int16(record_dict['agc_status_word']), + 'stat.lopwr': np.int16(record_dict['lp_status_word']), + # TODO: currently not implemented + 'noise.search': np.float32(0.0), + # TODO: currently not implemented + 'noise.mean': np.float32(0), + 'channel': np.int16(record_dict['slice_id']), + 'bmnum': np.int16(beam), + 'bmazm': np.float32(record_dict['beam_azms'][beam_index]), + 'scan': np.int16(record_dict['scan_start_marker']), + # no digital receiver offset or rxrise required in + # Borealis + 'offset': np.int16(0), + 'rxrise': np.int16(0), + 'intt.sc': np.int16(np.floor(record_dict['int_time'])), + 'intt.us': np.int32(np.fmod(record_dict['int_time'], 1.0) * 1e6), + 'txpl': np.int16(record_dict['tx_pulse_len']), + 'mpinc': np.int16(record_dict['tau_spacing']), + 'mppul': np.int16(len(record_dict['pulses'])), + # an alternate lag-zero will be given. + 'mplgs': np.int16(record_dict['lag_numbers'].shape[0] - 1), + 'nrang': np.int16(data_dimensions[1]), + 'frang': np.int16(round(record_dict['first_range'])), + 'rsep': np.int16(round(record_dict['range_sep'])), + # False if list is empty. + 'xcf': np.int16(bool('xcfs' in record_dict.keys())), + 'tfreq': np.int16(record_dict['freq']), + 'mxpwr': np.int32(-1), + 'lvmax': np.int32(20000), + 'rawacf.revision.major': np.int32(1), + 'rawacf.revision.minor': np.int32(0), + 'combf': 'Converted from Borealis file: ' + origin_string + + ' ; Number of beams in record: ' + str(len(record_dict['beam_nums'])) + + ' ; ' + record_dict['experiment_comment'][:].decode('utf-8') + + ' ; ' + record_dict['slice_comment'][:].decode('utf-8'), + 'thr': np.float32(0), + 'ptab': record_dict['pulses'].astype(np.int16), + 'ltab': record_dict['lag_pulses'].astype(np.int16), + 'pwr0': lag_zero_power.astype(np.float32), + # list from 0 to num_ranges + 'slist': np.array(list(range(0, data_dimensions[1]))).astype(np.int16), + 'acfd': correlation_dict['main_acfs'], + 'xcfd': correlation_dict['xcfs'] + } + record_dict_list.append(sdarn_record_dict) + + return record_dict_list diff --git a/pydarnio/dmap/__init__.py b/src/pydarnio/dmap/__init__.py similarity index 100% rename from pydarnio/dmap/__init__.py rename to src/pydarnio/dmap/__init__.py diff --git a/pydarnio/dmap/datastructures.py b/src/pydarnio/dmap/datastructures.py similarity index 100% rename from pydarnio/dmap/datastructures.py rename to src/pydarnio/dmap/datastructures.py diff --git a/pydarnio/dmap/dmap.py b/src/pydarnio/dmap/dmap.py similarity index 100% rename from pydarnio/dmap/dmap.py rename to src/pydarnio/dmap/dmap.py diff --git a/pydarnio/dmap/superdarn.py b/src/pydarnio/dmap/superdarn.py similarity index 100% rename from pydarnio/dmap/superdarn.py rename to src/pydarnio/dmap/superdarn.py diff --git a/pydarnio/dmap/superdarn_formats.py b/src/pydarnio/dmap/superdarn_formats.py similarity index 100% rename from pydarnio/dmap/superdarn_formats.py rename to src/pydarnio/dmap/superdarn_formats.py diff --git a/pydarnio/exceptions/__init__.py b/src/pydarnio/exceptions/__init__.py similarity index 100% rename from pydarnio/exceptions/__init__.py rename to src/pydarnio/exceptions/__init__.py diff --git a/pydarnio/exceptions/borealis_exceptions.py b/src/pydarnio/exceptions/borealis_exceptions.py similarity index 100% rename from pydarnio/exceptions/borealis_exceptions.py rename to src/pydarnio/exceptions/borealis_exceptions.py diff --git a/pydarnio/exceptions/dmap_exceptions.py b/src/pydarnio/exceptions/dmap_exceptions.py similarity index 100% rename from pydarnio/exceptions/dmap_exceptions.py rename to src/pydarnio/exceptions/dmap_exceptions.py diff --git a/pydarnio/exceptions/superdarn_exceptions.py b/src/pydarnio/exceptions/superdarn_exceptions.py similarity index 100% rename from pydarnio/exceptions/superdarn_exceptions.py rename to src/pydarnio/exceptions/superdarn_exceptions.py diff --git a/pydarnio/exceptions/warning_formatting.py b/src/pydarnio/exceptions/warning_formatting.py similarity index 100% rename from pydarnio/exceptions/warning_formatting.py rename to src/pydarnio/exceptions/warning_formatting.py diff --git a/pydarnio/utils/__init__.py b/src/pydarnio/utils/__init__.py similarity index 100% rename from pydarnio/utils/__init__.py rename to src/pydarnio/utils/__init__.py diff --git a/pydarnio/utils/conversions.py b/src/pydarnio/utils/conversions.py similarity index 100% rename from pydarnio/utils/conversions.py rename to src/pydarnio/utils/conversions.py