Added Sharpen Module (#6)

* Added Sharpen Module

* Updated block diagram, readme, user guide

* parameters tuning for Shar[en Filter

* Updated Results and Algo Description Document

* Minor error fixing in Readme

* RTL recommended changes made in AE and AWB

* Removed Linting Errors

README.md

@@ -33,7 +33,7 @@ ISP pipeline for `InfiniteISP_ReferenceModel v1.0`
 
 4. **Video Processing**: The model also features a video processing script that allows for sequential frame processing, with operational 3A Statistics data flowing between frames.
 
-
+5. **Support Multiple RAW formats**: The model supports supports renowned raw formats, for example, DNG (predominantly employed by Android devices), NEF (specific to Nikon devices), and CR2 (Canon's signature format). It can also process the .raw  image format, a one-dimensional pixel data array with no metadata header.
 
 ## Objectives
 
@@ -59,6 +59,7 @@ The table below provides a feature list of the model. The version `1.0` of the m
 | Gamma Correction             |Implements a LUT from config |
 | Auto Exposure                                 | [Auto Exposure](https://www.atlantis-press.com/article/25875811.pdf) <br> - AE stats calculations based on skewness |
 | Color Space Conversion                        | YCbCr digital <br> - BT 601 <br> - Bt 709  <br>   |YCbCr digital <br> - BT 601 <br> - Bt 709  <br> |
+| Edge Enhancement / Sharpeining                | Simple unsharp masking with strength control| 
 | Noise Reduction                               | [Non-local means filter](https://www.ipol.im/pub/art/2011/bcm_nlm/article.pdf) <br> - Implements intensity level difference  through a LUT|
 | RGB Conversion                               | Converts YCbCr digital image to RGB|
 | Invalid Region Crop    | Crops image to a fixed size|
@@ -103,14 +104,15 @@ RAW_DATA = './in_frames/normal/data'
 
 ## How to Run on Pipeline on Multiple Images/Dataset
 
-There are two scripts that run Infinite-ISP on multiple images:
+There is another script [isp_pipeline_multiple_images.py](isp_pipeline_multiple_images.py) that runs Infinite-ISP on multiple images with two modes:
 
 
-1. [isp_pipeline_dataset.py](isp_pipeline_dataset.py) 
+1. DATASET PROCESSING
     <br >Execute multiple images. Raw image should have its own config file with name `<filename>-configs.yml` where `<filename>` is raw filename otherwise the default configuration file [configs.yml](config/configs.yml) is used.
 
+    For raw image format such as, NEF, DNG and CR2 we have also provided a funcationality to extract sensor information provided in these raw files metadata and update default config file.
 
-2. [video_processing.py](video_processing.py) 
+2. VIDEO MODE
    <br> Each image in the dataset is considered as video frame in sequence. All images use the same configuration parameters from [configs.yml](config/configs.yml) and 3A Stats calculated on a frame are applied to the next frame.
 
 After cloning the repository and installing all the dependencies follow the following steps:
@@ -124,7 +126,7 @@ DATASET_PATH = './in_frames/normal/data'
 
 ```shell
 git submodule add <url> <path>
-git submodule update –-init –-recursive
+git submodule update --init --recursive
 ``` 
 
 
@@ -159,11 +161,11 @@ A comparison of the above results based on PSNR and SSIM image quality metrics
 
 | Images    | PSNR  | SSIM  |
 |-----------|-------|-------|
-| Indoor1   |22.5788     |0.8923
-|Outdoor1   |19.1544     |0.9048
-|Outdoor2   |18.8681     |0.8071
-|Outdoor3   |17.2825     |0.7304
-|Outdoor4   |19.9814     |0.8198
+| Indoor1   | 21.9826  |  0.8891
+|Outdoor1   | 17.7066   | 0.8959
+|Outdoor2   | 18.4617   | 0.8316
+|Outdoor3   | 15.9183   | 0.7351
+|Outdoor4   | 15.9183   | 0.7351
 
 ## User Guide
 

config/configs.yml

@@ -124,6 +124,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 5
+  sharpen_strength: 1
+  is_save: false
+
 2d_noise_reduction:
   is_enable: false
   window_size: 9

docs/UserGuide.md

@@ -153,7 +153,15 @@ Below parameters are present each ISP pipeline module they effect the functional
 | color_space_conversion | Details                                                                             |  
 |------------------------|------------------------------------------------------------------------------------                                |   
 | conv_standard          | The standard to be used for conversion <br> - `1` : Bt.709 HD <br> - `2` : Bt.601/407 |   
-
+
+### Edge Enchancement / Sharpening 
+
+| Sharpening         | Details                                           | 
+|--------------------|---------------------------------------------------|
+| is_enable           | When enabled applies the sharpening | 
+| sharpen_sigma      | Define the Standard Deviation of the Gaussian Filter |   
+| sharpen_strength   | Controls the sharpen strength applied on the high frequency components  |  
+
 ### 2d Noise Reduction
 
 | 2d_noise_reduction | Details                                           | 

in_frames/normal/data/Indoor1_2592x1536_10bit_GRBG-configs.yml

@@ -125,6 +125,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

in_frames/normal/data/Outdoor1_2592x1536_10bit_GRBG-configs.yml

@@ -124,8 +124,14 @@ auto_exposure:
 
 color_space_conversion:
   conv_standard: 2
-  is_save: False
-
+  is_save: false
+
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

in_frames/normal/data/Outdoor2_2592x1536_10bit_GRBG-configs.yml

@@ -126,6 +126,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

in_frames/normal/data/Outdoor3_2592x1536_10bit_GRBG-configs.yml

@@ -127,6 +127,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

in_frames/normal/data/Outdoor4_2592x1536_10bit_GRBG-configs.yml

@@ -126,6 +126,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

infinite_isp.py

@@ -24,6 +24,7 @@
 from modules.color_correction_matrix import ColorCorrectionMatrix as CCM
 from modules.color_space_conversion import ColorSpaceConversion as CSC
 from modules.yuv_conv_format import YUVConvFormat as YUV_C
+from modules.sharpen import Sharpening as SHARP
 from modules.noise_reduction_2d import NoiseReduction2d as NR2D
 from modules.rgb_conversion import RGBConversion as RGBC
 from modules.invalid_region_crop import InvalidRegionCrop as IRC
@@ -75,6 +76,7 @@ def load_config(self, config_path):
         self.parm_gmc = c_yaml["gamma_correction"]
         self.parm_ae = c_yaml["auto_exposure"]
         self.parm_csc = c_yaml["color_space_conversion"]
+        self.parm_sha = c_yaml["sharpen"]
         self.parm_2dn = c_yaml["2d_noise_reduction"]
         self.parm_rgb = c_yaml["rgb_conversion"]
         self.parm_irc = c_yaml["invalid_region_crop"]
@@ -187,9 +189,14 @@ def run_pipeline(self, visualize_output=True):
         csc = CSC(gamma_raw, self.platform, self.sensor_info, self.parm_csc)
         csc_img = csc.execute()
 
+        # =====================================================================
+        # Sharpening
+        sharp = SHARP(csc_img, self.platform, self.sensor_info, self.parm_sha)
+        sharp_img = sharp.execute()
+
         # =====================================================================
         # 2d noise reduction
-        nr2d = NR2D(csc_img, self.sensor_info, self.parm_2dn, self.platform)
+        nr2d = NR2D(sharp_img, self.sensor_info, self.parm_2dn, self.platform)
         nr2d_img = nr2d.execute()
 
         # =====================================================================

modules/auto_exposure.py

@@ -48,7 +48,7 @@ def apply_window_offset_crop(self):
         """
         Get AE Stats window by cropping the offsets
         """
-        offsets = np.ceil(self.stats_window_offset / 4) * 4
+        offsets = self.stats_window_offset
         top = int(offsets[0])
         bottom = None if offsets[1] == 0 else -int(offsets[1])
         left = int(offsets[2])
@@ -104,7 +104,7 @@ def get_luminance_histogram_skewness(self, img):
         Zwillinger, D. and Kokoska, S. (2000). CRC Standard Probability and Statistics
         Tables and Formulae. Chapman & Hall: New York. 2000. Section 2.2.24.1
         """
-
+        img_orig = np.copy(img)
         # First subtract central luminance to calculate skewness around it
         img = img.astype(np.float64) - self.center_illuminance
 
@@ -120,18 +120,29 @@ def get_luminance_histogram_skewness(self, img):
         if self.is_debug:
             print("   - AE - Actual_Skewness = ", skewness)
 
-        sign_m3 = np.sign(m_3)
-
-        m_2 = m_2 >> 6
-        m_3 = abs(m_3) >> 9
-
-        approx_sqrt_m_2 = approx_sqrt(m_2)
-        new_skewness, _ = get_approximate(m_3 / (m_2 * approx_sqrt_m_2), 16, 8)
-        new_skewness = sign_m3 * new_skewness
+        # all integer calc
+        img_int = img_orig.astype(np.int64) - self.center_illuminance
+        img_int_size = img_int.size
+        m_2_int = np.sum(np.power(img_int, 2)).astype(np.int64)
+        m_3_int = np.sum(np.power(img_int, 3)).astype(np.int64)
+        m_2_int = np.int64(m_2_int / img_int_size)
+        m_3_int = np.int64(m_3_int / img_int_size)
+        sign_m3_int = np.sign(m_3_int)
+        # all integer calc
+
+        m_2_int = m_2_int >> 6
+        m_3_int = abs(m_3_int) >> 9
+
+        approx_sqrt_m_2_int = approx_sqrt(m_2_int)
+        new_skewness_int = (
+            np.int64((m_3_int * 256) / (m_2_int * approx_sqrt_m_2_int)) / 256
+        )
+        new_skewness_int = sign_m3_int * new_skewness_int
         if self.is_debug:
-            print("   - AE - Approx_Skewness = ", new_skewness)
 
-        return new_skewness
+            print("   - AE - Approx_Skewness Int = ", new_skewness_int)
+
+        return new_skewness_int
 
     def execute(self):
         """

modules/auto_white_balance.py

@@ -34,7 +34,7 @@ def apply_window_offset_crop(self):
         """
         Get AWB Stats window by cropping the offsets
         """
-        offsets = np.ceil(self.stats_window_offset / 4) * 4
+        offsets = self.stats_window_offset
         top = int(offsets[0])
         bottom = None if offsets[1] == 0 else -int(offsets[1])
         left = int(offsets[2])

modules/sharpen.py

@@ -0,0 +1,109 @@
+"""
+File: sharpen.py
+Description: Simple unsharp masking with frequency and strength control.
+Code / Paper  Reference:
+Author: xx-isp ([email protected])
+"""
+
+import time
+import numpy as np
+from scipy import ndimage
+
+from util.utils import save_output_array_yuv
+
+
+class Sharpening:
+    """
+    Sharpening
+    """
+
+    def __init__(self, img, platform, sensor_info, parm_sha):
+        self.img = img
+        self.enable = parm_sha["is_enable"]
+        self.sensor_info = sensor_info
+        self.parm_sha = parm_sha
+        self.is_save = parm_sha["is_save"]
+        self.platform = platform
+
+    def gaussian_kernel(self, size_x, size_y=None, sigma_x=5, sigma_y=None):
+        """
+        Generate a Gaussian kernel for convolutions for Sharpening Algorithm
+        """
+        if size_y is None:
+            size_y = size_x
+        if sigma_y is None:
+            sigma_y = sigma_x
+
+        assert isinstance(size_x, int)
+        assert isinstance(size_y, int)
+
+        x_0 = size_x // 2
+        y_0 = size_y // 2
+
+        x_axis = np.arange(0, size_x, dtype=float)
+        y_axis = np.arange(0, size_y, dtype=float)[:, np.newaxis]
+
+        x_axis -= x_0
+        y_axis -= y_0
+
+        exp_part = x_axis**2 / (2 * sigma_x**2) + y_axis**2 / (2 * sigma_y**2)
+        return 1 / (2 * np.pi * sigma_x * sigma_y) * np.exp(-exp_part)
+
+    def apply_sharpen(self):
+        """Sharpens an image using the unsharp mask algorithm.
+
+        Args:
+            image: A numpy array of shape (height, width) representing the image to be sharpened.
+            kernel_size: The size of the Gaussian kernel to use for blurring the image.
+            sigma: The standard deviation of the Gaussian kernel.
+
+        Returns:
+            A numpy array of shape (height, width) representing the sharpened image.
+        """
+
+        sigma = self.parm_sha["sharpen_sigma"]
+        kernel_size = int(2 * np.ceil(2 * sigma) + 1)
+
+        kernel = self.gaussian_kernel(kernel_size, kernel_size, sigma, sigma)
+
+        kernel = (kernel * (2**20)).astype(np.int)
+
+        luma = (self.img[:, :, 0]).astype(np.int)
+        # Filter the luma component of the image with a Gaussian LPF
+        # Smoothing magnitude can be controlled with the sharpen_sigma parameter
+        smoothened = ndimage.correlate(luma, kernel, mode="mirror")
+        smoothened = smoothened / (2**20)
+
+        # Sharpen the image with upsharp mask
+        # Strength is tuneable with the sharpen_strength parameter
+        sharpened = luma + ((luma - smoothened) * self.parm_sha["sharpen_strength"])
+
+        self.img[:, :, 0] = np.uint8(np.clip(sharpened, 0, 255))
+        return self.img
+
+    def save(self):
+        """
+        Function to save module output
+        """
+        if self.is_save:
+            save_output_array_yuv(
+                self.platform["in_file"],
+                self.img,
+                "Out_Shapening_",
+                self.platform,
+            )
+
+    def execute(self):
+        """
+        Applying sharpening to input image
+        """
+        print("Sharpening = " + str(self.enable))
+
+        if self.enable is True:
+            start = time.time()
+            s_out = self.apply_sharpen()
+            print(f"  Execution time: {time.time() - start:.3f}s")
+            self.img = s_out
+
+        self.save()
+        return self.img