Cherrypick for rc3 (#1254)

* Fix logic to not align grism data in standard pipeline (#1243) * Pull changes from master in to build 3.4.0rc3 * Insure both FLT and FLC headers have same a posteriori fit keywords (#1238) * Update DRZ headers with same WCS keywords as DRC headers * Restore align_utils * Fixed problem in pipeline processing * Fix cut-and-paste error * update how make_poller_files.py generates visit numbers to account an edge case (#1221) * make_poller_files.py: updated how visit number column of .out files is determined. * diagnostic_utils.py: updated how the value for visit number in section "general information" is generated to protect against edge case * RickerWavelet Kernel for SBC needs to have dimensions which are odd. (#1246) Changed to 23 from 24. * Address issue of logging for PyTest tests (#1247) * Added better traceback information to the Exception in product.py. Added/renamed a pytest: test_svm_wcsonly.py to test_svm_canary.py and renamed its associated input file containing a list of filenames: test_acs_hrc_sbc_input.out to acs_hrc_sbc_input.out. * Modified the align.py module, perform_align() and determine_fit_quality(), by defining global variables to use for the log file handler and log filename. This was done to address the PyTests from having an error looking for a directory associated with the first test as all PyTests are done in a single Python session. The error would cause the test to fail, and the tests need to be executed to ensure the algorithms continue to work as expected. Essentially, the log file handler is reset as necessary so a fresh handle is assured for each new test. Some PyTests were updated to PASS. Updating the tests further is a topic for a different PR. * Changes done as a result of PR review: (1) PyTest updates to make some comparisons more efficient by using NumPy.isclose, and (2) try/except the os.remove() which could fail on Windows. There is no impact here for Operations as this would be on a Windows system. * Build process (#1244) fix build system * Remove logic from align (#1250) * Remove nictools as a dependency (#1245) * remove nictools as a dependency by moving the code into drizzlepac * remove unused import * MVM: Make tool to quantify quality of GAIA alignment generic for general use (#1241) * analyze_mvm_gaia_alignment.py: added subroutine 'apply_gaia_pm_correction' * analyze_mvm_gaia_alignment.py: Created subroutine find_and_match_sources() from lines 224-264 * analyze_mvm_gaia_alignment.py: reordered find_and_match_sources() inputs to be in a more logical order; Fleshed out find_and_match_sources() docstring 'Parameters' descriptions; Updated apply_gaia_pm_correction() docstring; Added for loop and logic to perform() to try to run find_and_match_sources() with custom-computed FWHM value and fail over to hard-coded value * analyze_mvm_gaia_alignment.py: Fixed misspelled words, PEP8 issues * analyze_mvm_gaia_alignment.py: removed unused subroutine apply_gaia_pm_correction(). * analyze_mvm_gaia_alignment.py: Added a few lines of code to report WCSNAME for each input image in perform() * analyze_mvm_gaia_alignment.py: Added a few lines of code to report WCSNAME for each input image in perform() * analyze_mvm_gaia_alignment.py: Fleshed out head-end docstring. * analyze_mvm_gaia_alignment.py: refactored commandline argument parsing to accept either a list of flc/flt files or a file containing a list of input flc/flt files * analyze_mvm_gaia_alignment.py: refactored commandline argument parsing to accept either a list of flc/flt files or a file containing a list of input flc/flt files * analyze_mvm_gaia_alignment.py: refactored perform() so that it can handle either a user-specified list of flc/flt files or a user-specified name of a text file containing a list of flc/flt files. * analyze_mvm_gaia_alignment.py: refactored perform() so that it can handle either a user-specified list of flc/flt files or a user-specified name of a text file containing a list of flc/flt files. * analyze_mvm_gaia_alignment.py: Added min_n_matches as user-defined input parameter to perform() as per reviewer's suggestion * analyze_mvm_gaia_alignment.py: tweaked the np.invert() call in step 4. * Fix incorrect drizzling of 0-weight input pixels (#1222) * Fix incorrect drizzling of 0-weight input pixels * Fix unit test * Refine headers for filter and total products (#1249) * Refine headers for filter and total products * Fix parameter naming bug * Fixed typos in rules files and docstring * Add tables as a dependency for the use of Pandas Co-authored-by: Michael Dulude <[email protected]> Co-authored-by: mdlpstsci <[email protected]> Co-authored-by: Nadia Dencheva <[email protected]> Co-authored-by: Mihai Cara <[email protected]>
spacetelescope · Jan 26, 2022 · 03a8865 · 03a8865
1 parent 62161bb
commit 03a8865
Show file tree

Hide file tree

Showing 77 changed files with 3,065 additions and 505 deletions.
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -20,9 +20,15 @@ of the list).
 
 - Detect extension name from WFPC2 flatfield files. [#1193]
 
+- Refactored the build system to be PEP-517 ad PEP-518 complient. [#1244]
+
+- Fixed a bug in the drizzle algorithm due to which input pixels with
+  zero weights may still contribute to the output image. [#1222]
+
 
 3.3.0 (28-Sep-2021)
 ===================
+
 This version includes all the functionality needed to generate
 source catalogs, both point source and extended (segment) source
 catalogs, during single-visit mosaic (SVM) processing.  In fact,

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -1,10 +1,8 @@
-include RELIC-INFO
 recursive-include doc *
 global-include *.py *.md *.notes *.yml *.toml
 global-include *.x *.c *.h *.par
 exclude */version.py
 prune .pytest_cache
 prune .eggs
 prune .git
-prune relic
 prune drizzlepac/htmlhelp
diff --git a/doc/source/conf.py b/doc/source/conf.py
@@ -18,7 +18,8 @@
 import sys
 import sphinx
 
-from distutils.version import LooseVersion
+from packaging.version import Version
+
 try:
     from ConfigParser import ConfigParser
 except ImportError:
@@ -55,8 +56,8 @@ def setup(app):
 
 
 def check_sphinx_version(expected_version):
-    sphinx_version = LooseVersion(sphinx.__version__)
-    expected_version = LooseVersion(expected_version)
+    sphinx_version = Version(sphinx.__version__)
+    expected_version = Version(expected_version)
     if sphinx_version < expected_version:
         raise RuntimeError(
             "At least Sphinx version {0} is required to build this "
@@ -105,7 +106,7 @@ def check_sphinx_version(expected_version):
 if on_rtd:
     extensions.append('sphinx.ext.mathjax')
 
-elif LooseVersion(sphinx.__version__) < LooseVersion('1.4'):
+elif Version(sphinx.__version__) < Version('1.4'):
     extensions.append('sphinx.ext.pngmath')
 
 else:
@@ -131,14 +132,10 @@ def check_sphinx_version(expected_version):
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
-#
-# The short X.Y version.
-# version = '1.0.6'
-from drizzlepac import __version__, __version_date__
-version = __version__
+from drizzlepac import __version__ as version
 # The full version, including alpha/beta/rc tags.
 # release = '1.0.6 (14-Aug-2012)'
-release = "{:s} ({:s})".format(__version__, __version_date__)
+release = version
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/drizzlepac/__init__.py b/drizzlepac/__init__.py
@@ -15,8 +15,26 @@
 
 """
 import os
+import re
+import sys
+from pkg_resources import get_distribution, DistributionNotFound
 
-from .version import *
+__version_commit__ = ''
+_regex_git_hash = re.compile(r'.*\+g(\w+)')
+
+try:
+    __version__ = get_distribution(__name__).version
+except DistributionNotFound:
+    __version__ = 'dev'
+
+if '+' in __version__:
+    commit = _regex_git_hash.match(__version__).groups()
+    if commit:
+        __version_commit__ = commit[0]
+
+
+#if sys.version_info < (3, 8):
+#    raise ImportError("Drizzlepac requires Python 3.8 and above.")
 
 from . import ablot
 from . import adrizzle

diff --git a/drizzlepac/ablot.py b/drizzlepac/ablot.py
@@ -27,7 +27,7 @@
     print('\n Please check the installation of this package to insure C code was built successfully.')
     raise ImportError
 
-from .version import *
+from . import __version__
 
 __all__ = ['blot', 'runBlot', 'help', 'getHelpAsString']
 
@@ -454,9 +454,7 @@ def getHelpAsString(docstring = False, show_ver = True):
 
     if docstring or (not docstring and not os.path.exists(htmlfile)):
         if show_ver:
-            helpString = os.linesep + \
-                ' '.join([__taskname__, 'Version', __version__,
-                ' updated on ', __version_date__]) + 2*os.linesep
+            helpString = f"\n{__taskname__} Version {__version__}\n"
         else:
             helpString = ''
         if os.path.exists(helpfile):

diff --git a/drizzlepac/adrizzle.py b/drizzlepac/adrizzle.py
@@ -18,7 +18,7 @@
 import stwcs
 from stwcs import distortion
 
-from .version import *
+from . import __version__
 
 try:
     from . import cdriz
@@ -1195,9 +1195,7 @@ def getHelpAsString(docstring=False, show_ver=True):
 
     if docstring or (not docstring and not os.path.exists(htmlfile)):
         if show_ver:
-            helpString = os.linesep + \
-                ' '.join([__taskname__, 'Version', __version__,
-                ' updated on ', __version_date__]) + 2 * os.linesep
+            helpString = f"\n{__taskname__} Version {__version__}\n"
         else:
             helpString = ''
         if os.path.exists(helpfile):

diff --git a/drizzlepac/align.py b/drizzlepac/align.py
@@ -32,7 +32,6 @@
 SPLUNK_MSG_FORMAT = '%(asctime)s %(levelname)s src=%(name)s- %(message)s'
 
 __version__ = 0.0
-__version_date__ = '21-Aug-2019'
 
 def _init_logger():
     log = logutil.create_logger(__name__, level=logutil.logging.NOTSET, stream=sys.stdout,
@@ -41,6 +40,10 @@ def _init_logger():
 
 log = _init_logger()
 
+# Initial values for the module log filename and the associated file handler used for the log
+module_fh = None
+module_logfile = ""
+
 # ----------------------------------------------------------------------------------------------------------
 
 
@@ -166,7 +169,7 @@ def check_and_get_data(input_list: list, **pars: object) -> list:
 # ------------------------------------------------------------------------------------------------------------
 def perform_align(input_list, catalog_list, num_sources, archive=False, clobber=False, debug=False,
                   update_hdr_wcs=False, result=None,
-                  runfile=None, print_fit_parameters=True, print_git_info=False, output=False,
+                  runfile="temp_align.log", print_fit_parameters=True, print_git_info=False, output=False,
                   headerlet_filenames=None, fit_label=None, product_type=None,
                   **alignment_pars):
     """Actual Main calling function.
@@ -243,20 +246,25 @@ def perform_align(input_list, catalog_list, num_sources, archive=False, clobber=
         Table which contains processing information and alignment results for every raw image evaluated
 
     """
-    log.info("*** HAP PIPELINE Processing Version {!s} ({!s}) started at: {!s} ***\n".format(__version__, __version_date__, util._ptime()[0]))
-
     if debug:
         loglevel = logutil.logging.DEBUG
     else:
         loglevel = logutil.logging.INFO
 
-    if runfile is not None:
-        loglevel = logutil.logging.DEBUG
-        fh = logutil.logging.FileHandler(runfile)
-        fh.setLevel(loglevel)
-        log.addHandler(fh)
+    # Need to ensure the logging works properly for the PyTests where each test starts with a fresh handler
+    global module_fh
+    global module_logfile
+    if module_fh is not None:
+        print("Removing old file handler for logging.")
+        log.removeHandler(module_fh)
+
+    module_logfile = runfile.upper()
+    module_fh = logutil.logging.FileHandler(runfile)
+    module_fh.setLevel(loglevel)
 
+    log.addHandler(module_fh)
     log.setLevel(loglevel)
+    log.info(f"{__taskname__} Version {__version__}\n")
 
     # 0: print git info
     if print_git_info:
@@ -354,8 +362,7 @@ def perform_align(input_list, catalog_list, num_sources, archive=False, clobber=
             index = np.where(alignment_table.filtered_table['imageName'] == imgname)[0][0]
 
             # First ensure sources were found
-
-            if table is None or not table[1]:
+            if table is None:
                 log.warning("No sources found in image {}".format(imgname))
                 alignment_table.filtered_table[:]['status'] = 1
                 alignment_table.filtered_table[:]['processMsg'] = "No sources found"
@@ -473,7 +480,9 @@ def perform_align(input_list, catalog_list, num_sources, archive=False, clobber=
                                 alignment_table.filtered_table,
                                 (catalog_index < (len(catalog_list) - 1)),
                                 apars,
-                                print_fit_parameters=print_fit_parameters)
+                                print_fit_parameters=print_fit_parameters,
+                                loglevel=loglevel,
+                                runfile=runfile)
                         alignment_table.filtered_table = filtered_table
 
                         # save fit algorithm name to dictionary key "fit method" in imglist.
@@ -642,7 +651,8 @@ def make_label(label, starting_dt):
 # ----------------------------------------------------------------------------------------------------------
 
 
-def determine_fit_quality(imglist, filtered_table, catalogs_remaining, align_pars, print_fit_parameters=True):
+def determine_fit_quality(imglist, filtered_table, catalogs_remaining, align_pars, print_fit_parameters=True,
+                          loglevel=logutil.logging.NOTSET, runfile="temp_align.log"):
     """Determine the quality of the fit to the data
 
     Parameters
@@ -672,6 +682,13 @@ def determine_fit_quality(imglist, filtered_table, catalogs_remaining, align_par
     print_fit_parameters : bool
         Specify whether or not to print out FIT results for each chip
 
+    log_level : int, optional
+        The desired level of verboseness in the log statements displayed on the screen and written to the
+        .log file. Default value is 20, or 'info'.
+
+    runfile : string
+        log file name
+
     Returns
     -------
     max_rms_val : float
@@ -700,6 +717,25 @@ def determine_fit_quality(imglist, filtered_table, catalogs_remaining, align_par
             * fit compromised status (Boolean)
             * reason fit is considered 'compromised' (only populated if "compromised" field is "True")
     """
+
+    # Set up the log file handler and name of the log file
+    # If the log file handler were never set, the module_fh will be None.
+    # Only want to remove a file handler if there were one set in the first place.
+    global module_fh
+    global module_logfile
+    #if module_fh is not None and module_logfile != runfile.upper():
+    if module_fh is not None:
+        print("Removing old file handler for logging.")
+        log.removeHandler(module_fh)
+
+    module_logfile = runfile.upper()
+    module_fh = logutil.logging.FileHandler(runfile)
+    module_fh.setLevel(loglevel)
+
+    log.addHandler(module_fh)
+    log.setLevel(loglevel)
+    log.info("Log file: {}".format(module_logfile))
+
     max_rms_val = 1e9
     num_xmatches = 0
     fit_status_dict = {}
@@ -919,7 +955,7 @@ def determine_fit_quality(imglist, filtered_table, catalogs_remaining, align_par
 # ----------------------------------------------------------------------------------------------------------------------
 
 def determine_fit_quality_mvm_interface(imglist, filtered_table, catalogs_remaining, ref_catalog_length,
-                                        align_pars, print_fit_parameters=True, loglevel=logutil.logging.NOTSET):
+                                        align_pars, print_fit_parameters=True, loglevel=logutil.logging.NOTSET, runfile="temp_align.log"):
     """Simple interface to allow MVM code to use determine_fit_quality().
 
     Parameters
@@ -949,6 +985,13 @@ def determine_fit_quality_mvm_interface(imglist, filtered_table, catalogs_remain
     print_fit_parameters : bool
         Specify whether or not to print out FIT results for each chip
 
+    log_level : int, optional
+        The desired level of verboseness in the log statements displayed on the screen and written to the
+        .log file. Default value is 20, or 'info'.
+
+    runfile : string
+        log file name
+
     Returns
     -------
     is_good_fit : bool
@@ -992,7 +1035,9 @@ def determine_fit_quality_mvm_interface(imglist, filtered_table, catalogs_remain
                                                                                            filtered_table,
                                                                                            catalogs_remaining,
                                                                                            align_pars,
-                                                                                           print_fit_parameters)
+                                                                                           print_fit_parameters=print_fit_parameters,
+                                                                                           loglevel=loglevel,
+                                                                                           runfile=runfile)
 
     # Determine if the fit quality is acceptable
     if fit_quality in align_pars['determine_fit_quality']['GOOD_FIT_QUALITY_VALUES']:

diff --git a/drizzlepac/astrodrizzle.py b/drizzlepac/astrodrizzle.py
@@ -45,7 +45,7 @@
 from . import staticMask
 from . import util
 from . import wcs_functions
-from .version import *
+from . import __version__
 
 
 __taskname__ = "astrodrizzle"
@@ -169,8 +169,8 @@ def run(configobj, wcsmap=None):
     clean = configobj['STATE OF INPUT FILES']['clean']
     procSteps = util.ProcSteps()
 
-    print("AstroDrizzle Version {:s} ({:s}) started at: {:s}\n"
-          .format(__version__, __version_date__, util._ptime()[0]))
+    print("AstroDrizzle Version {:s} started at: {:s}\n"
+          .format(__version__, util._ptime()[0]))
     util.print_pkg_versions(log=log)
 
     log.debug('')
@@ -294,9 +294,7 @@ def getHelpAsString(docstring = False, show_ver = True):
 
     if docstring or (not docstring and not os.path.exists(htmlfile)):
         if show_ver:
-            helpString = os.linesep + \
-                ' '.join([__taskname__, 'Version', __version__,
-                ' updated on ', __version_date__]) + 2*os.linesep
+            helpString = f"\n{__taskname__} Version {__version__}\n"
         else:
             helpString = ''
         if os.path.exists(helpfile):

diff --git a/drizzlepac/buildwcs.py b/drizzlepac/buildwcs.py
@@ -21,7 +21,6 @@
 
 # This is specifically NOT intended to match the package-wide version information.
 __version__ = '0.1.1'
-__version_date__ = '13-July-2020'
 
 # These default parameter values have the same keys as the parameters from
 # the configObj interface
@@ -428,9 +427,7 @@ def getHelpAsString(docstring = False, show_ver = True):
 
     if docstring or (not docstring and not os.path.exists(htmlfile)):
         if show_ver:
-            helpString = os.linesep + \
-                ' '.join([__taskname__, 'Version', __version__,
-                ' updated on ', __version_date__]) + 2*os.linesep
+            helpString = os.linesep + ' '.join([__taskname__, 'Version', __version__]) + os.linesep
         else:
             helpString = ''
         if os.path.exists(helpfile):

diff --git a/drizzlepac/catalogs.py b/drizzlepac/catalogs.py
@@ -6,10 +6,8 @@
 """
 import os, sys
 import copy
-from distutils.version import LooseVersion
 
 import numpy as np
-#import pywcs
 import astropy
 from astropy import wcs as pywcs
 import astropy.coordinates as coords

diff --git a/drizzlepac/createMedian.py b/drizzlepac/createMedian.py
@@ -22,7 +22,7 @@
 from . import processInput
 from .adrizzle import _single_step_num_
 
-from .version import *
+from . import __version__
 
 # look in drizzlepac for createMedian.cfg:
 __taskname__ = "drizzlepac.createMedian"
@@ -509,8 +509,7 @@ def getHelpAsString(docstring=False, show_ver=True):
     if docstring or (not docstring and not os.path.exists(htmlfile)):
         if show_ver:
             helpString = os.linesep + \
-                ' '.join([__taskname__, 'Version', __version__,
-                ' updated on ', __version_date__]) + 2*os.linesep
+                ' '.join([__taskname__, 'Version', __version__]) + os.linesep
         else:
             helpString = ''