- dkist.net.transfer_complete_datasets will now only create one Globus task for all datasets it downloads. (#340)
- Add a FileManager property to TiledDataset for tracking files more easily. (#437)
- Improve performance of computing arrays when using non-thread or sync dask schedulers, by reducing the amount of data needed to be sent to each worker. (#455)
- Fix a bug with dkist.net.transfer_complete_datasets where a length one
UnifiedResponsewould cause an error. (#340) - Fix mark argument parsing in the new pytest hook. (#440)
- Minor tweak to correct indexing of >4D datasets. (#453)
- Add --ds and --tiled-ds CLI options to allow passing in datasets for use with the test suite. The dataset given with --ds is passed to tests marked with accept_cli_dataset and those given with --tiled-ds are passed to tests marked with accept_cli_tiled_dataset. (#439)
- Minor updates to TiledDataset.plot() for working with more complex arrangements of tiles. (#441)
- Fix failure to fetch CLI options in publish build. (#447)
- Add various features for easier inspection of TiledDataset: - __repr__ method to output basic dataset info; - tiles_shape property to access data array shape for each individual tile; - slice_tiles() method to apply the same slice to all datasets. (#402)
- Add TiledDataset.plot() quicklook method. (#408)
- Add CircleCI config to enable figure comparison testing with pytest_mpl. (#415)
- Add some more benchmarks to track performance of more parts of the user tools. (#387)
- Add benchmarks for slicing and computing datasets. (#394)
- Added two partial datasets to dkist.data.sample for documentation and testing. (#421)
- replace usages of
copy_arrayswithmemmapforasdf>=3.1.0(#422) - Update Dataset representation for better readability. (#431)
- SKip a failing test in the release publish build. (#434)
- Add "status" to the list of know dataset inventory fields. (#417)
- Catch URLError when trying to download attr values in tests so that the existing file isn't assumed to be corrupted and therefore deleted. (#407)
- Update ASDF schemas for upcoming ASDF standard 1.6.0. (#324)
- Improve performance of
VaryingCelestialTransformclasses by not creating a new transform for every set of parameters but instead update the parameters on a single model. (#370) - Add GitHub workflow and dependencies for Codspeed, to benchmark PRs against main. (#382)
- Run plotting benchmarks fewer times for more manageable CI, and add a benchmark for generate_celestial_transform. (#383)
- Fix broadcasting issues during pixel -> world conversion for models with a Ravel component. (#309)
- Fix a performance regression when dask>=2024.2.1 is installed. (#361)
- Add a how to guide describing how to reproject VBI data. Also migrate tutorial to the latest DDT datasets. (#349)
- Refactor various subclasses of VaryingCelestialTransform to centralise the calculations in preparation for improving performance. (#344)
- Correct Fido time searching to use endTimeMin and startTimeMax (in the correct order) so that searching returns any dataset with a partially or completely overlapping time range. (#336)
- Adjust file loading to support single-frame datasets with no time axis. (#335)
- Call the DKIST search API to automatically determine valid data search parameters and register those with the Fido client. (#311)
- Use a new feature in the DKIST datasets API to search for all datasets which intersect the given time. (#326)
- Fix some small issues with the installation instructions. (#323)
- Fix some deprecation warnings for Python 3.12 support. (#322)
- Add a logging framework to present information to users in a nicer way.
The logger can be accessed as
dkist.logto change log levels etc. (#317)
- Bump minimum version of asdf to 2.11.2 to pick up jsonschema bugfix. (#313)
- Change the
appdirsdependency for the maintainedplatformdirspackage. (#318) - Fix an unpinned minimum version of
asdf-wcs-schemascausing potential read errors on newest asdf files with dkist 1.1.0. (#320)
- We now require gwcs 0.19+ and therefore astropy 5.3+ (#305)
- Add a new
AsymmetricMappingmodel to allow a different mapping in the forward and reverse directions. (#305)
- Fix the oversight where, when generating a model for a celestial WCS, the scale model was put before the affine transform in the pipeline. This means that the units for the affine transform matrix provided to
VaryingCelestialTransformandgenerate_celestial_transformshould be pixels not degrees. (#305) - Fix missing references to parent transform schemas in
RavelandVaryingCelestialTransformASDF schemas. (#305)
- To improve compatibility with external libraries that provide ASDF serialization and validation (like asdf-astropy) dkist schemas were updated to use tag wildcards when checking tagged objects (instead of requiring specific tag versions). (#308)
- The ASDF files currently being served by the data center are incompatible with gwcs 0.19+. This is due to a change in how Stokes coordinates are represented. In this release we have pinned the gwcs version to <0.19. A future release will require 0.19+ when the ASDF files have been updated. (#301)
- Add missing dependencies to setup.cfg - explicit is better than implicit. (#294)
- Import ValidationError from asdf, drop jsonschema as a dependency. (#295)
- Implement missing
select_tagmethod ofDatasetConverter. (#297) - Update varying celestial transform schema ref to use a uri instead of a tag. (#298)
- Ensure that we don't nest Dask arrays when no FITS files can be read. This might result in more memory being used when computing an array with missing files. (#301)
- Add a new dkist.load_dataset function to combine and replace
Dataset.from_directory()andDataset.from_asdf(). (#274) - Add the ability to load more than one asdf file at once to dkist.load_dataset. (#287)
- Fix minor bugs for header slicing functionality and expand test coverage for edge-cases. (#275)
- Fixed inverse transform in .VaryingCelestialTransformSlit2D. Which fixes a bug in VISP WCSes. (#285)
- Fix a bug preventing the transfer of a single dataset with :meth:`~dkist.net.transfer_complete_datasets`. (#288)
- Added a new tutorial section based on the NSO workshop material. (#281)
- Add jsonschema as an explicit dependency (previously it was provided by asdf). (#274)
- Update minimum required versions of asdf, asdf-astropy, dask, matplotlib, numpy, parfive, and sunpy. (#275)
- Add path interpolation to :meth:`~dkist.net.transfer_complete_datasets` path location argument. (#266)
- Add a .Dataset.inventory attribute to more easily access the inventory metadata (previously
.meta['inventory']. (#272) - Add experimental support for 3D LUTs to
TimeVaryingCelestialTransformclasses. (#277)
- Improve speed of
import dkistby preventing automatic import ofdkist.net. (#266) - Fix how Fido uses Wavelength to search for datasets. (#266)
- Fix using
a.dkist.Embargoed.falseanda.dkist.Embargoed.trueto specify embargo status. (#266) - Add units support to
a.dkist.FriedParameter. (#266) - Add search attrs corresponding to new columns in dataset inventory. (#266)
- Make dkist.Dataset return the appropriately sliced header table when slicing data. (#271)
- Update docstring for :meth:`dkist.net.transfer_complete_datasets` to include previously missing
pathparameter. (#273)
- Adds support to Ravel for N-dimensional data. (#249)
- Add support for passing a list of dataset IDs as strings to dkist.net.transfer_complete_datasets. (#240)
- Fix errors with some types of input in dkist.net.transfer_complete_datasets. (#240)
- Fix searching for Globus endpoints with SDK 3 (#240)
- Fixes bug in the inverse property of CoupledCompoundModel by correcting the various mappings in the inverse. (#242)
- Drop support for Python 3.8 in line with NEP 29. (#232)
- Add new methods :meth:`.FileManager.quality_report` and :meth:`.FileManager.preview_movie` to download the quality report and preview movie. These are accessed as
Dataset.files.quality_reportandDataset.files.preview_movie. (#235)
- Unit for
lon_polewas set to the spatial unit of the input parameters within ~dkist.wcs.models.VaryingCelestialTransform. It is now fixed to always be degrees. (#225) - Add a new model to take a 2D index and return the corresponding correct index for a 1D array, and the inverse model for the reverse operation. To be used as a compound with Tabular1D so that it looks like a Tabular2D but the compound model can still be inverted. (#227)
- Internal improvements to how the data are loaded from the collection of FITS files. This should have no user facing effects, but provides a foundation for future performance work. (#232)
- Add ability to page through the DKIST results and affect the page size. (#212)
- Fix, and make required, the unit property on a dataset in ASDF files. (#221)
- Fix bugs in testing caused by the release of
pytest 7.2.0. (#210) - Make loading a mosaiced VBI dataset work with
Dataset.from_asdf. (#213) - Add support for Python 3.11 (#218)
- Add documentation for available path interpolation keys. (#207)
- Add a
label=kwarg to .FileManager.download and dkist.net.transfer_complete_datasets allowing the user to completely customise the Globus transfer task label. (#193)
- Successfully ask for re-authentication when Globus token is stale. (#197)
- Fix a bug where
FileManager.downloadwould fail if there was not an asdf file or quality report PDF in inventory. (#199) - Fix an issue with slicing a dataset where the slicing wouldn't work correctly if the first axis of the data array has length one. (#199)
- No more invalid characters in default Globus label name. (#200)
- Hide extraneous names in dkist.net.attrs with underscores so they don't get imported when using that module. (#201)
- Catch empty return value from data search in dkist.net.transfer_complete_datasets and raise a
ValueErrortelling the user what's happening. (#204)
- Support passing a whole ~sunpy.net.fido_factory.UnifiedResponse to ~dkist.net.transfer_complete_datasets. (#165)
- Support pretty formatting of new Dataset Inventory fields in Fido search results table. (#165)
- Refactor .FileManager to correctly support slicing. (#176)
- Unify path handling between .FileManager.download and .DKISTClient.fetch. This means that you can use the same path specification to download the ASDF files and the FITS files, using keys such as "Dataset ID". (#178)
- Use the new
/datasets/v1/configendpoint to automatically retrieve the globus endpoint ID corresponding to the dataset searcher in use. (#136) - Add a new function dkist.net.transfer_complete_datasets which takes a single row from a
Fidosearch or a dataset ID and sets up a Globus transfer task for the complete dataset. (#136) - Migrate to Globus SDK version 3+. Also use the config system to configure endpoints for dataset search and metadata download. (#136)
- Rename
dkist.net.DKISTDatasetClienttodkist.net.DKISTClient. The only user facing change this has is to modify the key used when slicing the return fromFido.search. (#136)
- Implement models where the pointing varies along the second pixel axis (for rastering slit spectrographs). (#161)
- Fix behaviour of VaryingCelestialTransform when called with arrays of pixel or world coordinates. (#160)
- Implement Astropy models to support spatial transforms which change with a third pixel axis. (#148)
- Add ASDF serialization for VaryingCelestialTransform and CoupledCompoundModel. (#156)
- Fix asdf using old schema and tag versions when saving new files. (#157)
- Migrate to the asdf 2.8+
Converterinterface, this bumps various dependencies but should have no effect on reading or writing asdf files. (#152)
- The inventory record and the headers table are now both stored in the
Dataset.metadict rather than headers being it's own attribute. This means it is more likely to be carried through correctly when doing operations designed forNDCubeobjects. (#139) - Add support for tiled datasets in the spatial dimensions. This adds a new class dkist.TiledDataset which holds a 2D grid of dkist.Dataset objects, and associated asdf schemas to serialise them. (#143)
Move file handling and download tooling onto .Dataset.files, which is now a pointer to a class which has all the information to generate the arrays.
Also the loaders generated by the new .FileManager class now have a reference to the .FileManager which generated them, which means that the basepath can be dynamically generated by reference. (#126)
Modify the dkist.io.FileManager class so that most of the functionality exists in the new base class and the download method is in the separate child class. In addition make more of the API private to not confuse end users. (#130)
- Write initial guide to the user tools and tidy up the API docs (#127)
- Fix a bug where sometimes the path wouldn't be set correctly after FITS file download. (#124)
- Fix display of sliced datasets in repr and correctly propagate slicing operations to the array container. (#119)
- Implement .DKISTClient.fetch to download asdf files from the metadata streamer service. (#90)
- Enable tests on Windows (#95)
- Added search bounding box functionality to DKIST client. (#100)
- Added support for new dataset search parameters (
hasSpectralAxis,hasTemporalAxis,averageDatasetSpectralSamplingMin,averageDatasetSpectralSamplingMax,averageDatasetSpatialSamplingMin,averageDatasetSpatialSamplingMax,averageDatasetTemporalSamplingMin,averageDatasetTemporalSamplingMax) (#108)
- Support gwcs 0.14 and ndcube 2.0.0b1 (#86)
- Update Fido client for changes in sunpy 2.1; bump the sunpy dependency to at least 2.1rc3. (#89)
- Move asdf generation code into dkist-inventory package (#79)
- Move the
dkist.asdf_makerpackage todkist.io.asdf.generatorwhile also refactoring its internal structure to hopefully make it a little easier to follow. (#71)
- Add dkist.Dataset class to represent a dataset to the user. (#1)
- Add code for converting a nested list of asdf.ExternalArrayReference objects to a dask.array.Array. (#1)
- Add implementation of
Dataset.pixel_to_worldandDataset.world_to_pixel. (#1) - Add ability to crop Dataset array by world coordinates. (#1)
- Add a reader for asdf files. (#1)
- Add a dkist config file using custom location from astropy (#3)
- Add functionality for making asdf files from collections of calibrated FITS files. (#10)
- Python 3.6+ Only. (#11)
- Add framework for slicing gwcses. (#18)
- Implement dataset slicing. This orders the results of WCS related methods on the dataset class in reverse order to that of the underlying WCS. So it is not so jarring that the array and WCS are in reverse order. (#20)
- Add a
dataset_from_fitsfunction that generates an asdf file in a directory with a set of FITS files. (#21) - Add support for array wcs calls post slicing a non-separable dimension. (#23)
- Add
relative_tokwarg todkist.asdf_maker.generator.dataset_from_fitsanddkist.asdf_maker.generator.asdf_tree_from_filenames. (#26) - Add support for 2D plotting with WCSAxes. (#27)
- All asdf files are now validated against the level 1 dataset schema on save and load. (#41)
- Add support for returning an array of NaNs when the file is not present. This is needed to support partial dataset download from the DC. (#43)
- Add utilities for doing OAuth with Globus. (#46)
- Add helper functions for listing a globus endpoint (#49)
- Add support for multiple globus oauth scopes (#50)
- Added support for starting and monitoring Globus transfer tasks (#55)
- Allow easy access to the filenames contained in an
dkist.io.BaseFITSArrayContainerobject via a.filenamesproperty. (#56) dkist.io.BaseFITSArrayContainerobjects are now sliceable. (#56)- Initial implementation of
dkist.Dataset.downloadmethod for transferring files via globus (#57) - Rely on development NDCube 2 for all slicing and plotting code (#60)
- Change Level 1 asdf layout to use a tag and schema for
Dataset. This allows reading of asdf files independent from the dkist.Dataset class. (#66) - Implement a new more efficient asdf schema and tag for
BaseFITSArrayContainerto massively improve asdf load times. (#70) - Add a sunpy.net.Fido client for searching DKIST Dataset inventory. Currently only supports search. (#73)
- Implement correct extraction of dataset inventory from headers and gwcs. Also updates some data to be closer to the in progress outgoing header spec (214) (#76)
- Fix the units in
spatial_model_from_header(#19) - Correctly parse headers when generating gwcses so that only values that change along that physical axis are considered. (#21)
- Reverse the ordering of gWCS objects generated by
asdf_helpersas they are cartesian ordered not numpy ordered (#21) - Fix incorrect compound model tree splitting when the split needed to happen at the top layer (#23)
- Fix a lot of bugs in dataset generation and wcs slicing. (#24)
- Fix incorrect chunks when creating a dask array from a loader_array. (#26)
- Add support for dask 2+ and make that the minimum version (#68)
- Migrate the dkist.Dataset class to use gWCS's APE 14 API (#32)