part1_lightfield.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:orange\">HW6 - Part 1 : All about lightfields</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is part 1 of homework 6. In this homework we will dive deeper into the concept of how lightfield works.\n",
    "\n",
    "We have already discussed lightfields in detail in class, but we acknoledge that the concept of a 4D-lightfield might a bit hard to understand. However, it is one of the most important concepts in Computational Photography and we think it's important for you to understand this better.\n",
    "\n",
    "We promise - HW6 might be a bit lengthly, but it's a lot of fun and very instructive. For this homework we've tried to remove all non-necesseary functions (like plotting and data loading), so that you don't have to implement them, which should save a lot of time. The functions that we have implemented for you can be found in util_lightfield.py and util_synthetic.py. You need to implement the functions in lightfield.py and synthetic_aperture.py."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand the concept of light-field better, we show here to example images which show what a light-field is and one application. The application shown here is for calculating a 3D-scene from a light-field."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://graphics.stanford.edu/projects/array/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![test](pictures/lightfield-aquisition.jpg)\n",
    "\n",
    "From: http://limu.ait.kyushu-u.ac.jp/e/project/image/0003a.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![test](pictures/lightfield-aquisition2.jpg)\n",
    "\n",
    "From: http://limu.ait.kyushu-u.ac.jp/e/project/image/0003a.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:red\">Writing Tasks</span>\n",
    "\n",
    "Since the coding part of this assignment is a little longer, we decided to only ask for a short writeup this time. In particular, we will not ask you to answer specific questions about the implemented methods in your writeup. Some questions are posed in between the different coding tasks, but they should be seen more as a guidance. Written answers to these questions are NOT mandatory.\n",
    "\n",
    "Your writeup for this assignment should be 1-2 pages max and should consist of the following sections:\n",
    "\n",
    "**Abstract:** 3-4 Sentences motivating light field imaging\n",
    "\n",
    "**Introduction:** Write a concise introduction to the topic. Possible points to discuss are: What is a lightfield? Why is it importat in computational imaging?How is it acquired? Name a few aplications of lightfields.\n",
    "\n",
    "**(Methods:)** This section is OPTIONAL. If you feel that a method you implemented should be discussed in detail (e.g., because you found a very creative solution), feel free to write this down here.\n",
    "\n",
    "**Results:** Show a few of your greatest results. Describe what the reader sees and emphasize the important points. Exlain why the reults you picked are so cool.\n",
    "\n",
    "**Conclusion:** A very short summary of what you've learned and what you think about the assignment.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Information on autoreload: https://ipython.org/ipython-doc/3/config/extensions/autoreload.html\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from __future__ import print_function\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import scipy\n",
    "import scipy.misc\n",
    "from matplotlib import gridspec\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as patches\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "import skimage\n",
    "import skimage.morphology\n",
    "import skimage.transform as skimage_transform\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import src.code as code\n",
    "from src import synthetic_aperture\n",
    "import src.lightfield as lightfield\n",
    "\n",
    "import src.util_lightfield as util_lightfield"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:orange\">Problem 1: Loading Data of perfect lightfield</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're implementing all the loading data and dataset functions for you. We hope that those work out-of-the box for your system. If not, please post on campuswire so that we can find a solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"data//treasure_chest//\" # You can load a dataset later if you want :)\n",
    "\n",
    "path = \"data//chess//\" # for now we're working with the chess-dataset\n",
    "\n",
    "# You can use this to read the files\n",
    "files = glob.glob(path + \"*.png\")\n",
    "files = sorted(files) # Make sure to sort the file-list otherwise you might get probems\n",
    "files[0:10] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now load the data. This is traversing through a bunch of files. So have some patients. Every file that is loaded should be printed, so you can check how far you are in loading the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = util_lightfield.load_dataset(path,0.5) # 0.5 is for the scaling factor. We don't want to work on the full resolution images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's explore our data slighlty\n",
    "print(data.shape) # If everything went well this should be: (17, 17, 400, 700, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_,_,num_x,num_y,_ = data.shape\n",
    "print(num_x)\n",
    "print(num_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 1: Visualize cropped versions of the light-field</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot_image_grid function is aleady implemented for you.\n",
    "You can play around it's parameter to learn more about the light-field! please do so to familarize yourself more with what data we are dealing here with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,10))\n",
    "\n",
    "x0 = int(num_x/4)\n",
    "y0 = int(num_y/4)\n",
    "\n",
    "x1 = int(num_x*3.0/4)\n",
    "y1 = int(num_y*3.0/4)\n",
    "\n",
    "m_plots = 5\n",
    "n_plots = 5\n",
    "\n",
    "util_lightfield.plot_images_grid(data,m_plots, n_plots,x0,x1,y0,y1)\n",
    "plt.tight_layout()\n",
    "\n",
    "util_lightfield.save_fig_as_png('lightfield_cropped')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 2: Visualize the complete light-field</span>\n",
    "\n",
    "There's nothing to implement for you. Just use the \".plot_images_grid\" function and look at your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will show the complete light-field\n",
    "plt.figure(figsize=(15,12))\n",
    "util_lightfield.plot_images_grid(data,10,10)\n",
    "plt.tight_layout()\n",
    "\n",
    "util_lightfield.save_fig_as_png('lightfield_all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:orange\">Task: Explore a bit of the light-field interactively with this viewer</span>\n",
    "\n",
    "There is nothing for you to implement here. The image overview from above is just hard to interprete. We want to make it a bit easier for you understand how this light-field actually looks like.\n",
    "\n",
    "HINT: If you experience some weird big jumps (small jumps are normal), this might be a hint that your image loader might be wrong. Probably something with the indices went wrong."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HINT: TO use this interactive viewer you need the package ipywidgets.\n",
    "Here's some guide how you can install it: https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
    "\n",
    "If you have further problems with installing it, please write on campuswire for assistance. This is can be a bit trick to get installed correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(u,v):\n",
    "    plt.figure(figsize=(15,10))\n",
    "    print(u,v)\n",
    "    myim = data[u,v,:,:,:]\n",
    "    \n",
    "    plt.imshow(myim)\n",
    "    plt.show()\n",
    "\n",
    "x_widget =  widgets.IntSlider(min=0, max=data.shape[0]-1, step=1)\n",
    "y_widget = widgets.IntSlider(min=0, max=data.shape[1]-1, step=1)\n",
    "    \n",
    "interactive_plot = interactive(f, u = x_widget, v = y_widget)\n",
    "output = interactive_plot.children[-1]\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:orange\">Problem 1: Bilinear Interpolation to interpolate the light-field</span>\n",
    "\n",
    "The light-field has a fairly large spatial resolution, however the angular resolution is only $17 \\times 17$. That means we can only jump into views that are exactly captured at the specific camera positions.\n",
    "\n",
    "However, using bilinear inerpolation we can simulate views from arbitrary positions. Bilinear interpolation is a very powerful image-processing technique which is well implemented in openCV and other packages. However, those packages don't work with 4D-lightfields. \n",
    "\n",
    "In the next cells you will implement your own bilinear-interpolation code. We'll start with implementation in 2D and slowly improve your code until you are able to implement 4D-lightfields."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A good introduction to **bilinear-interpolation** is provided by the wikipedia page on: https://en.wikipedia.org/wiki/Bilinear_interpolation\n",
    "        \n",
    "You don't have to read the whole article, since we're already extracting the most important formula for you:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center> $ f(x,y) =  f(Q_{11})(x_2 - x)(y_2 - y)  + f(Q_{21})(x - x_1)(y_2 - y) + f(Q_{12})(x_2 - x)(y - y_1) + f(Q_{22})(x - x_2)(y - y_1)$</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we're here ommitting the factor $\\frac{1}{(x_2 - x_1)\\cdot(y_2 - y1)}$\n",
    "\n",
    "**Why can we do this?** Think about what $x_2$ is in our case. It's defined as $x_2 = x_1 + 1$ since we are working on a regular grid. Hence the differnce will always be 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"pictures/bilinear_interpolation.png\">\n",
    "\n",
    "Figure 1: The four red dots show the data points and the green dot is the point at which we want to interpolate. (from Wikipedia)\n",
    "\n",
    "<img src=\"pictures/Bilininterp.png\" style=\"width:300px;\">\n",
    "\n",
    "Figure 2: Example of bilinear interpolation on the unit square with the z values 0, 1, 1 and 0.5 as indicated. Interpolated values in between represented by color."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: This might be on the hardest challenge you'll have to face in 331 in terms when it comes to trickiness in implementation and working with array broadcasting etc.\n",
    "\n",
    "Luckily, we're not interest in interpolating in spatial coordinates, we're only interested in upsampling the angular components. So this makes it a bit easier. No, worries, we'll go through what you have to do step-by-step!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 1: Let's create a sample image in 2D</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a very easy 2x2 matrix. We want to interpolate the values in-between\n",
    "my_test_img = np.array([[1,2],[3,4]])\n",
    "plt.imshow(my_test_img)\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 2: Implement Bilinear interpolation for test image</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have to implement the bilinear interpolation for this test image. \n",
    "\n",
    "Here's an interpolation very close to what we need: \n",
    "https://gist.github.com/peteflorence/a1da2c759ca1ac2b74af9a83f69ce20e\n",
    "( which is taken from this discussion https://stackoverflow.com/questions/8661537/how-to-perform-bilinear-interpolation-in-python )\n",
    "\n",
    "NOTE: This implementation will not work completely. There are still some slight changes that you'll need to perform, but it's a very good starting point.\n",
    "\n",
    "I recommend copying this code over in a cell and trying it out line-by-line. It's really important that you understand what each line in this code is doing and WHY it is needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_test_img.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create the values where we want to interpolate\n",
    "N_new = 3\n",
    "xx = np.linspace(0,1,N_new)\n",
    "[X,Y] = np.meshgrid(xx,xx)\n",
    "\n",
    "out = lightfield.bilinear_interpolate_numpy_matrix(my_test_img,X,Y)\n",
    "plt.imshow(out)\n",
    "\n",
    "util_lightfield.save_fig_as_png('bilinear_matrix_3x3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's do the same for N_new = 100 (should look like a very smooth image)\n",
    "N_new = 50\n",
    "xx = np.linspace(0,1,N_new)\n",
    "[X,Y] = np.meshgrid(xx,xx)\n",
    "\n",
    "out = lightfield.bilinear_interpolate_numpy_matrix(my_test_img,X,Y)\n",
    "plt.imshow(out)\n",
    "\n",
    "util_lightfield.save_fig_as_png('bilinear_matrix_50x50')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 2: Implement Bilinear interpolation for test image</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now test if your algorithm still works for 2D images. Copy your code from \"bilinear_interpolate_numpy_matrix\"\n",
    "to \"bilinear_interpolate_numpy_2D_image\".\n",
    "\n",
    "If implemented correctly, this should work right away. If not, adapt it until it works. We create a second function, so that you're not losing your implementation and you can start over at any time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = skimage.data.camera()\n",
    "\n",
    "img_down = skimage_transform.rescale(img,0.25)\n",
    "\n",
    "xx = np.linspace(0,img_down.shape[0],img.shape[0])\n",
    "yy = np.linspace(0,img_down.shape[1],img.shape[0])\n",
    "\n",
    "[X,Y] = np.meshgrid(xx,yy)\n",
    "\n",
    "img_up = lightfield.bilinear_interpolate_numpy_2D_image(img_down,X,Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use this code to visualize if this worked well\n",
    "\n",
    "plt.figure(figsize=(15,10))\n",
    "\n",
    "plt.subplot(131)\n",
    "plt.imshow(img,cmap='gray')\n",
    "plt.title(\"Original\")\n",
    "\n",
    "plt.subplot(132)\n",
    "plt.imshow(img_down,cmap='gray')\n",
    "plt.title(\"Downsampled\")\n",
    "\n",
    "plt.subplot(133)\n",
    "plt.imshow(img_up,cmap='gray')\n",
    "plt.title(\"Upsampled\")\n",
    "\n",
    "\n",
    "util_lightfield.save_fig_as_png('bilinear_cameraman')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 3: Modify Bilinear interpolation to work with arbitrary color images with arbitrary shapes</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we go one step further. Instead of gray-scale images that are of square size, you now have to adapt your method, so that it workes for RGB-images that are rectangular.\n",
    "\n",
    "Copy your implementation from \"bilinear_interpolate_numpy_2d_image\" to \"bilinear_interpolate_numpy_RGB_image\" and modify it until it works. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = skimage.data.rocket()\n",
    "\n",
    "scale = 0.1\n",
    "\n",
    "dw = skimage_transform.rescale(img[:,:,0],scale).shape\n",
    "img_down = np.zeros((dw[0],dw[1],3))\n",
    "\n",
    "\n",
    "for k in range(0,3):\n",
    "    img_down[:,:,k] = skimage_transform.rescale(img[:,:,k],scale)\n",
    "\n",
    "# We have to cast it back to uint8\n",
    "img_down = (255*img_down).astype(np.uint8)\n",
    "    \n",
    "plt.imshow(img_down)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test your bilinear upsampler "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xx = np.linspace(0,img_down.shape[0],img.shape[0])\n",
    "yy = np.linspace(0,img_down.shape[1],img.shape[1])\n",
    "\n",
    "[X,Y] = np.meshgrid(xx,yy)\n",
    "\n",
    "img_up = lightfield.bilinear_interpolate_numpy_RGB_image(img_down,X,Y)\n",
    "\n",
    "print(img_up.shape)\n",
    "\n",
    "plt.figure(figsize=(15,10))\n",
    "\n",
    "plt.subplot(131)\n",
    "plt.imshow(img)\n",
    "plt.title(\"Original\")\n",
    "\n",
    "plt.subplot(132)\n",
    "plt.imshow(img_down)\n",
    "plt.title(\"Downsampled\")\n",
    "\n",
    "plt.subplot(133)\n",
    "plt.imshow(img_up)\n",
    "plt.title(\"Upsampled\")\n",
    "\n",
    "util_lightfield.save_fig_as_png('bilinear_matrix_rocket')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 4: Test Bilinear interpolation of one light-field pixel-image</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now comes the most complicated part:\n",
    "\n",
    "Our lightfield data has the following datastructure: (N_u,N_v,N_x,N_z,N_col)\n",
    "\n",
    "where\n",
    " 1. N_u and N_v are the angular components (those are small)\n",
    " 2. N_x and N_y are the spatial dimensins (those are large)\n",
    " 3. N_col should be 3 for 3 colors\n",
    " \n",
    "Copy your code from \"bilinear_interpolate_numpy_RGB_image\" over to \"bilinear_interpolate_numpy\". Your implementation will be starting point for interpolating the light-field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just for reminding us of the light-field dimensions\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's start out with subsampling a simple RGB image\n",
    "# This should work without any problem\n",
    "\n",
    "x = 150\n",
    "y = 550\n",
    "sub_data = data[:,:,x,y,:]\n",
    "print(sub_data.shape)\n",
    "\n",
    "plt.imshow(sub_data)\n",
    "plt.title(\"Light-Field Data at Pixel \" + str(x) + \" | \" + str(y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create an array that goes from 0 to 17 (min to max) with 100 points to sample from in between\n",
    "xx = np.linspace(0,16,100)\n",
    "\n",
    "plt.plot(xx)\n",
    "\n",
    "[X,Y] = np.meshgrid(xx,xx)\n",
    "\n",
    "lf_up = lightfield.bilinear_interpolate_numpy(sub_data,X,Y)\n",
    "\n",
    "print(lf_up.shape) \n",
    "# This should now be an array of dimension (100,100,3).\n",
    "# We have upsampled our lightfield images from 17x17 to 100x100.\n",
    "# That means instead of 17x17 views we now have 100^2 views (10k which is quite a lot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If this has worked and run through correctly (which it should since you've already implemented interpolation for RGB images above). The following code-cell should show you the interpolate lightfield for one image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,10))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.imshow(sub_data)\n",
    "plt.title(\"Low-Resolution Light-Field Image\")\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.imshow(lf_up)\n",
    "plt.title(\"Upsampled light-field image\")\n",
    "\n",
    "\n",
    "util_lightfield.save_fig_as_png('bilinear_lightfield_2d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 5: Bilinear interpolation for all light-field pixel-image</span>\n",
    "\n",
    "This will be the trickiest part ! We now need to perform this interpolation for each spatial pixel ( we have a lot of those !!!)! With a 1Mpx light-field (the original data are actually larger! those are already 1 Million interpolations).\n",
    "\n",
    "Since we are writing in Python, we cannot use 2 for-loops for this because it would become incredibly slow. Fortunately, numpy vector-math (using broadcasting) is doing this magic for us. You only have to lern how to this!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll start with a small subset of only a 2x4 images (we use 2 x 4 to have different dimnsions for you to test)\n",
    "# We also change the numbers in the lightfield for u and v direction to make sure we don't mess up the order of dimensions!\n",
    "\n",
    "sub_data = data[0:15,0:14,150:152,550:554,:]\n",
    "\n",
    "print(sub_data.shape)\n",
    "\n",
    "xx = np.linspace(0,16,32) # \n",
    "yy = np.linspace(0,15,30) # \n",
    "\n",
    "[X,Y] = np.meshgrid(xx,yy)\n",
    "\n",
    "lf_up = lightfield.bilinear_interpolate_numpy(sub_data,X,Y)\n",
    "print(lf_up.shape) # The last 3 dimensions shoudld be the same as sub_data.shape !!!\n",
    "# this shou'd be (32, 30, 2, 4 ,3) according to the dimensions that we've chosen above.\n",
    "# If you choose e.g. 36 points to interpolate instead of 34 it should change accordingly.\n",
    "# Make sure that this works well before you move on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,10))\n",
    "\n",
    "plt.subplot(221)\n",
    "plt.imshow(sub_data[:,:,0,0,:].squeeze())\n",
    "plt.title(\"Low-Resolution Light-Field Image\")\n",
    "\n",
    "plt.subplot(222)\n",
    "plt.imshow(lf_up[:,:,0,0,:].squeeze())\n",
    "plt.title(\"Upsampled light-field image\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now that you've implemented this for sub part, check if it also works for a larger lightfield!\n",
    "\n",
    "# We're now interpolating the 4D-lightfield along the u-v axis. However, we're not doing it for the complete image yet, but only for a subsection\n",
    "sub_data = data[:,:,100:152,500:554,:]\n",
    "\n",
    "print(sub_data.shape)\n",
    "\n",
    "xx = np.linspace(0,17,34)\n",
    "\n",
    "[X,Y] = np.meshgrid(xx,xx)\n",
    "\n",
    "lf_up = lightfield.bilinear_interpolate_numpy(sub_data,X,Y)\n",
    "\n",
    "print(lf_up.shape) # The last 3 dimensions shoudld be the same as sub_data.shape !!!\n",
    "\n",
    "plt.figure(figsize=(12,10))\n",
    "\n",
    "plt.subplot(131)\n",
    "plt.imshow(sub_data[0,0,:,:,:].squeeze())\n",
    "plt.title(\"Low-Resolution Light-Field Image\")\n",
    "\n",
    "# Please realizs that the lf_up at position 1|1 is an interpolated image (!) and not a captured dataset\n",
    "plt.subplot(132)\n",
    "plt.imshow(lf_up[1,1,:,:,:].squeeze())\n",
    "plt.title(\"Interpolated light-field image\")\n",
    "\n",
    "# Please realizs that the lf_up at position 2|2 is a measured image again!\n",
    "plt.subplot(133)\n",
    "plt.imshow(lf_up[2,2,:,:,:].squeeze())\n",
    "plt.title(\"Interpolated light-field image\")\n",
    "\n",
    "# This image should look almost the same, but it would be slighlty shifted in one direction. \n",
    "\n",
    "util_lightfield.save_fig_as_png('bilinear_matrix_lightfield_chess')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 7: Use your bilinear interpolation function to write an interactive light-field interpolator</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No worries, we have implemented the interactive light-field viewer for you! As long as your \"bilinear_interpolate_numpy\" works, you're in good shape!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(m, b):\n",
    "    plt.figure(figsize=(15,10))\n",
    "    \n",
    "    myim = lightfield.bilinear_interpolate_numpy(data[:,:,:,:,:],m,b)\n",
    "    \n",
    "    print(myim.shape)\n",
    "    \n",
    "    plt.imshow(myim)\n",
    "    plt.show()\n",
    "\n",
    "x_widget =  widgets.FloatSlider(min=0, max=data.shape[0]-1, step=0.25)\n",
    "y_widget = widgets.FloatSlider(min=0, max=data.shape[1]-1, step=0.25)\n",
    "    \n",
    "interactive_plot = interactive(f, m = x_widget, b = y_widget)\n",
    "output = interactive_plot.children[-1]\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:orange\">Problem 2: Digital Refocusing</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will no explore how we can use the lightfield to digitally refocus \n",
    "\n",
    "The original implementation comes from Ren Ng et al. [1] which was later implemented in a product calle Lytro, which was also founded by Ren Ng [2].\n",
    "\n",
    "I recommend giving those 2 papers a quick read. They are very informative and should help developping a much better understanding of light-fields.\n",
    "\n",
    "References\n",
    "\n",
    "[1] http://graphics.stanford.edu/papers/CameraArray/CameraArray_Sig05.pdf\n",
    "[2] https://graphics.stanford.edu/papers/lfcamera/lfcamera-150dpi.pdf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 1: Refocus in 1D</span>\n",
    "\n",
    "Before we're diving into refocusing our lightfield, we want to implement everything on 1D-signals which we extract from our lightfield. This will help us build a much better intuition what we'll have to do in 2D later on. Conceptually it is the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we extract 3 different columns from our lightfield dataset and choose one v-coorindate of the lightfield.\n",
    "y = [320, 350,380]\n",
    "v = 8\n",
    "\n",
    "# Let us visualize this data so that we get an understanding what data we'll be working with now\n",
    "plt.imshow(data[0,0,:,:])\n",
    "plt.plot([y,y],[0,data.shape[2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will extract the 1D-lightfield for each of the 3 lines. In total we will have 17 light-field positions per pixel. There is nothing for you to implement. Just look at the code so that you understand it and then try to analyze the pictures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize = (12,5))\n",
    "for k in range(len(y)):\n",
    "    plt.subplot(len(y),1,k+1)\n",
    "    img = data[:,v,:,y[k],0].squeeze()\n",
    "    plt.imshow(img,interpolation=\"bilinear\", aspect=\"auto\",cmap='gray')\n",
    "    plt.title(\"Lightfield for x = \" + str(y[k]) + \" | v = \" + str(v))\n",
    "plt.tight_layout()\n",
    "\n",
    "util_lightfield.save_fig_as_png('bilinear_1d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:red\">Suggestions for Report: Explain the 1D-Lightfield</span>\n",
    "\n",
    "Can you explain these 3 images? Try to interprete what you see in these images!\n",
    "\n",
    "Answer the following questions, if there is more that you think that is remarkable in the light-fields, feel free to write about it as well:\n",
    "\n",
    "1. Why are the values changing between black and gray ? \n",
    "2. Between pixel ~130 to ~200 these 3 image are signficiantly different. The At x = 320 it's mostluy black, at x = 350 you don't see anything and at x = 380 it's mostly silver. Why do you this happens ?\n",
    "3. Why does this look like a cone? Can you explain why these lines have different slopes?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 1: Refocus in 1D</span>\n",
    "\n",
    "We will now only work with one v-y slice which is at located at x = 350 because it has clear differences between black and white. This will make debugging and understanding of our lightfield refocusing the easiest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the data we want\n",
    "y = [350]\n",
    "v = 8\n",
    "\n",
    "img = data[0,0,:,:]\n",
    "\n",
    "# Visualize the data\n",
    "\n",
    "# We are doing this for you. Feel free to play around with this and visualize different light-field coordinates.\n",
    "# This is not a must, but it can very instructive for people who want to dig in a bit more!\n",
    "\n",
    "plt.figure(figsize=(10,3))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.ylim([0,data.shape[2]])\n",
    "plt.plot([y,y],[0,data.shape[2]])\n",
    "plt.imshow(img)\n",
    "plt.gca().invert_yaxis()\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.imshow(data[:,v,:,y[0]],interpolation=\"bilinear\", aspect=\"auto\",cmap='gray')\n",
    "\n",
    "plt.tight_layout()\n",
    "\n",
    "util_lightfield.save_fig_as_png('bilinear_1d_example')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 2: Explore the image in 1D-interactive view</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we extract only one 1D-lightfield image. Let's visualize quickly\n",
    "img = data[:,v,:,y[0],0]\n",
    "plt.figure(figsize=(10,2))\n",
    "plt.imshow(img,interpolation=\"bilinear\", aspect=\"auto\",cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#####  <span style=\"color:orange\">Subtask 2: 1D-interactive view</span>\n",
    "We have implemented an interactive 1D light-field viewer for you. Please play around with the slider to visualize your 1D-lightfield. There is no coding or writing task, just analze it so that you develop a better understanding of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(u):\n",
    "    plt.figure(figsize=(10,4))\n",
    "    \n",
    "    plt.subplot(211)\n",
    "    plt.plot(img[u,:])\n",
    "    plt.title(\"V-idx: \" + str(u))\n",
    "    plt.xlim(0,img.shape[1]-1)\n",
    "    \n",
    "    plt.subplot(212)\n",
    "    plt.imshow(img,interpolation=\"bilinear\", aspect=\"auto\",cmap='gray')\n",
    "    plt.plot([0,img.shape[1]],[u,u])\n",
    "    plt.xlim(0,img.shape[1])\n",
    "        \n",
    "    plt.show()\n",
    "\n",
    "u_widget =  widgets.IntSlider(min=0, max=img.shape[0]-1, step=1)\n",
    "    \n",
    "interactive_plot = interactive(f, u = u_widget)\n",
    "output = interactive_plot.children[-1]\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### <span style=\"color:orange\">Subtask 2: Look at the mean and indivudal lines</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the following cell. There is nothing suprising there, it should just help you a bit better in understanding our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,6))\n",
    "\n",
    "plt.subplot(211)\n",
    "plt.plot(img[::5,:].T)\n",
    "plt.xlim(0,img.shape[1])\n",
    "plt.title(\"Pinhole images\")\n",
    "plt.grid()\n",
    "\n",
    "plt.subplot(212)\n",
    "plt.plot(img.mean(axis=0))\n",
    "plt.xlim(0,img.shape[1])\n",
    "plt.grid()\n",
    "plt.title(\"mean image\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:orange\">Problem: Understand the refousing formula in 1D</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This formula is taken from Ren Ng's paper on lightfield refocusing. Instead of working in 2D, we're here having a 1D-version of the lightfield for simpler understanding\n",
    "\n",
    "<h1><center>$E(s') = \\int L \\left( u,s' - ( 1 - \\alpha ) \\cdot u  \\right) du$</center></h1>\n",
    "\n",
    "Here $\\alpha$ represents the refocus paramter (i.e. this encodes the focus distance). $L$ is the lightfield which is saved in our \"img\" variable. $u$ is the coordinate where the camera is shifted at and $s$ is the location on the detector.\n",
    "\n",
    "How does this look in a discretized version? Instead of an integral we now have a discretized sum over $u_k$.\n",
    "\n",
    "<h1><center>$E(s') = \\sum\\limits_k L \\left( u_k,s' - ( 1 - \\alpha ) \\cdot u_k  \\right)$</center></h1>\n",
    "\n",
    "If you look at this formula a bit longer, you will realize that his is nothing than adding several images together. However, each of this images is now slightly shifted, by the value $(1-\\alpha)\\cdot u$ which depends on the position $u$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 1: Calculate the amount of pixel shift</span>\n",
    "\n",
    "Now you will have to implement the get_shift_1D function that will give you for each u-light field coordinate the amount the 1D-signal needs to be shifted.\n",
    "\n",
    "The amount that needs to be shifted can be easily extracted from the formula above as\n",
    "\n",
    "$$ (1 - \\alpha)\\cdot u_k $$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = 0.3\n",
    "\n",
    "shifts = lightfield.get_shift_1D(img,alpha)\n",
    "    \n",
    "print(shifts.shape) # Should be vector with 17 entries\n",
    "\n",
    "plt.plot(shifts)\n",
    "plt.grid()\n",
    "plt.title(r\"$\\alpha$ = \" + str(alpha))\n",
    "plt.xlabel(\"Lightfield Coordinate u\")\n",
    "plt.ylabel(\"Amount of Pixel Shift\")\n",
    "\n",
    "util_lightfield.save_fig_as_png('shift_1d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 2: Shift all 1D-Signals according to the calculated shifts</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next step we'll have to actually implement the image shift/translation. \n",
    "\n",
    "shift_images_1d tkae the img and the shift that you've just calculated and shifts the images accordingly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_shifted = lightfield.shift_images_1d(img,shifts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(14,3))\n",
    "plt.subplot(121)\n",
    "plt.imshow(img,interpolation=\"bilinear\", aspect=\"auto\",cmap='gray')\n",
    "plt.title(\"Original Image\")\n",
    "plt.subplot(122)\n",
    "plt.imshow(img_shifted,interpolation=\"bilinear\", aspect=\"auto\",cmap='gray')\n",
    "plt.title(\"Image Shifted\")\n",
    "\n",
    "util_lightfield.save_fig_as_png('image_shifted')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 3: Average the 1D-Signal</span>\n",
    "\n",
    "Now all that is to do is to average the shifted image along the u-direction and you'll obtain the refocused 1D-signal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Implement the function average_1d_signal\n",
    "img_1d_refocused = lightfield.average_1d_signal(img_shifted)\n",
    "\n",
    "print(img_1d_refocused.shape)\n",
    "\n",
    "plt.figure(figsize=(10,2))\n",
    "util_lightfield.plot_1D_signal(img_1d_refocused,alpha)\n",
    "\n",
    "\n",
    "util_lightfield.save_fig_as_png('average_1d_signal')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 4: Play around with digital refocusing</span>\n",
    "\n",
    "The following cell-block is a small interactive tool that allows you to refocus to different depths using the slider value. If you've implemented everything well, this should work out of the box. The following functions should have been implemented by you now:\n",
    "1. get_shift_1d\n",
    "2. shift_images_1d\n",
    "3. average_1d_signal\n",
    "\n",
    "Play around with it and try to develop a feeling how the refocusing algorithms works in 1D. It will be very similair in 2D-case, however thinking in 4D is slightly more complex than in 2D, but conceptually it's pretty much the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(u):\n",
    "    plt.figure(figsize=(10,4))\n",
    "    \n",
    "    shifts = lightfield.get_shift_1D(img,u)\n",
    "    img_shifted = lightfield.shift_images_1d(img,shifts)\n",
    "    img_1d_refocused = lightfield.average_1d_signal(img_shifted)\n",
    "\n",
    "    \n",
    "    plt.subplot(211)\n",
    "\n",
    "    plt.imshow(img_shifted,interpolation=\"bilinear\", aspect=\"auto\",cmap='gray')\n",
    "\n",
    "    \n",
    "    plt.subplot(212)\n",
    "    \n",
    "    util_lightfield.plot_1D_signal(img_1d_refocused,u)\n",
    "    plt.tight_layout()\n",
    "\n",
    "\n",
    "u_widget =  widgets.FloatSlider(value=1,min=-2, max=3, step=0.05)\n",
    "\n",
    "interactive_plot = interactive(f, u = u_widget)\n",
    "output = interactive_plot.children[-1]\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 5: Extract some meaninful $\\alpha$-values</span>\n",
    "\n",
    "Play around with the interactive widget and write down a few values for $\\alpha$ where the image focuses on different part of the 1D-signal. \n",
    "\n",
    "A focused-signal is defined here where you see a sharp edge in the signal. What does the sharp edge represent in this signals ?\n",
    "\n",
    "HINT: Think about that we're looking at at chessboard, now what does black and gray values represent here?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define alpha values that focus on different parts in the image\n",
    "\n",
    "alpha_idxs = lightfield.get_alpha_values()\n",
    "print(type(alpha_idxs)) # Should be a list or a nunmpy array. Should have 3-5 different alpha positions that focus on different parts in the 1D image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task: Run the following block which shows how to estimate DoF</span>\n",
    "\n",
    "The code below is a very simple approximative way to estimate the depth-of-field of our signal.\n",
    "\n",
    "The main idea is to calculate the gradient (1D-derivtive) in x-direction of the shifted-image and then average over the lightfield coordinate u. \n",
    "\n",
    "Now, you should see some peaks whenever there was an edge in the original image. However, because we average the peaks should only be pronounced where the the line is perpendicular, otherwise the peaks will be blurred out. A blurred out peak corresponds to a signal that is outside the depth-of-field.\n",
    "\n",
    "In this code, we apply a simple peak-finding algorithm where we use the prominence of the peak as a classificator if the image is in or out-of-focus.\n",
    "\n",
    "There is nothing to implement for you. You can simply run this block and you'll see how it works. This code is also wrapped into the function lightfield.calculate_depth_of_field which we're using below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shifts = lightfield.get_shift_1D(img,alpha_idxs[0])\n",
    "img_shifted = lightfield.shift_images_1d(img,shifts)\n",
    "\n",
    "\n",
    "gradient_image = np.abs(np.gradient(img_shifted)[1])\n",
    "    \n",
    "plt.figure(figsize=(10,3))\n",
    "\n",
    "plt.subplot(311)\n",
    "plt.imshow(img_shifted,interpolation=\"bilinear\", aspect=\"auto\",cmap='gray')\n",
    "plt.subplot(312)\n",
    "plt.imshow(gradient_image,interpolation=\"bilinear\", aspect=\"auto\",cmap='gray')\n",
    "\n",
    "\n",
    "mean_gradient = np.mean(gradient_image,axis=0)\n",
    "\n",
    "plt.subplot(313)\n",
    "plt.plot(mean_gradient)\n",
    "\n",
    "plt.xlim(0,img_shifted.shape[1])\n",
    "\n",
    "peaks_all, _ = scipy.signal.find_peaks(mean_gradient, prominence=10, width=5)\n",
    "peaks_width, _ = scipy.signal.find_peaks(mean_gradient, prominence=10)\n",
    "\n",
    "peaks = util_lightfield.listComplementElements(peaks_width, peaks_all)\n",
    "peaks = np.array(peaks)\n",
    "\n",
    "plt.plot(peaks, mean_gradient[peaks], \"x\")\n",
    "\n",
    "img_refocused = lightfield.average_1d_signal(img_shifted)\n",
    "\n",
    "plt.plot([peaks[0],peaks[-1]],[0,0],linewidth=5)\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task: Visualize the refocused signal at different locations</span>\n",
    "\n",
    "Now run the following block where we have prepared some code that will plot the refocused 1D-signals depending on the alpha-values that you've chosen in your get_alpha_values() method.\n",
    "\n",
    "This image should look so that clearly see the focusing effect at 4 different image locations.\n",
    "\n",
    "The orange line roughly defines the region where the 1D-image is focused."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,8))\n",
    "\n",
    "for k in range(len(alpha_idxs)):\n",
    "    plt.subplot(len(alpha_idxs),1,k+1)\n",
    "    shifts = lightfield.get_shift_1D(img,alpha_idxs[k])\n",
    "    img_shifted = lightfield.shift_images_1d(img,shifts)\n",
    "    \n",
    "    img_refocused = lightfield.average_1d_signal(img_shifted)\n",
    "    \n",
    "    util_lightfield.plot_1D_signal(img_refocused,\"YOU FIGURE IT OUT\") # Replace this text with alpha_idxs[k]\n",
    "    \n",
    "    p0,p1 = util_lightfield.calculate_field_of_depth(img_shifted)\n",
    "    \n",
    "    plt.plot([p0,p1],[0,0],linewidth=5)\n",
    "\n",
    "plt.tight_layout()\n",
    "    \n",
    "    \n",
    "util_lightfield.save_fig_as_png('lightfield_refocus_1d_different_alpha')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:orange\">Problem: Refocusing in 4D-lightfield space or 2D-images</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've understood how refocusing works in 1D, we'll move over to digital refocusing in 2D."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 1 - Understand the orientation properties of the lightfield</span>\n",
    "\n",
    "The following code should help you analyze the lightfield a bit better. In particular we want you to understand how the lightfield coorindate u and v are oriented. Those will define in which direction you have to shift your images later on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the following code\n",
    "\n",
    "plt.figure(figsize = (15,10) )\n",
    "\n",
    "plt.subplot(221)\n",
    "\n",
    "plt.imshow(data[0,0])\n",
    "\n",
    "plt.title(\"0 | 0\")\n",
    "\n",
    "plt.subplot(222)\n",
    "\n",
    "plt.imshow(data[-1,0])\n",
    "\n",
    "plt.title(\"- 1 | 0\")\n",
    "\n",
    "plt.subplot(223)\n",
    "\n",
    "plt.imshow(data[0,-1])\n",
    "plt.title(\"0 | - 1\")\n",
    "\n",
    "plt.subplot(224)\n",
    "\n",
    "plt.imshow(data[-1,-1])\n",
    "plt.title(\"- 1 | - 1\")\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read more here:\n",
    "https://graphics.stanford.edu/courses/cs348b-competition/cs348b-14/second_report.pdf\n",
    "\n",
    "This is the formula that we have to implement. First, we write down the continuous formula, then the discretized version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><center>$E(s',t') = \\int L(u,v,\\frac{1}{\\alpha}(s' - ( 1 - \\alpha ) \\cdot u ), \\frac{1}{\\alpha} ( t' - (1 - \\alpha ) \\cdot v ) ) du dv$</center></h1>\n",
    "\n",
    "<br>\n",
    "and here in discretized version:\n",
    "\n",
    "<h1><center>$E(s',t') = \\sum\\limits_{k,p} L(u_k,v_p,\\frac{1}{\\alpha}(s' - ( 1 - \\alpha ) \\cdot u_k ), \\frac{1}{\\alpha} ( t' - (1 - \\alpha ) \\cdot v_p) )$</center></h1>\n",
    "\n",
    "If you compare this to the 1D-version, this is very similair. However, now you have shifts in both directions which you need to calculate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now, just like you did it for the 1D-signal, you will have to calculate the amount of shhift for a given alpha.\n",
    "# HINT: Since you're not working with 1D-signals, you are now working with 2D-meshgrid\n",
    "# Hence, your putput of the get_shifts positions should be a mehsgrid too. One meshgrid for the u coordinate, \n",
    "# and one for the v coordinate.\n",
    "\n",
    "\n",
    "alpha = 0.5\n",
    "\n",
    "DX, DY = lightfield.get_shifts(data.shape,alpha)\n",
    "\n",
    "# The following plots should help you analyze if you have implemented the light-field shift function correctly:\n",
    "\n",
    "plt.figure(figsize=(15,3))\n",
    "\n",
    "plt.subplot(131)\n",
    "plt.imshow(np.sqrt(DX**2 + DY**2))\n",
    "plt.colorbar()\n",
    "\n",
    "plt.subplot(132)\n",
    "plt.imshow(DX)\n",
    "plt.colorbar()\n",
    "plt.title(r\"DX($\\alpha=0.5$) | bottom (-4), top(+4)\")\n",
    "\n",
    "\n",
    "plt.subplot(133)\n",
    "plt.imshow(DY)\n",
    "plt.colorbar()\n",
    "plt.title(r\"DY($\\alpha=0.5$) | left (-4), right (+4)\")\n",
    "\n",
    "\n",
    "util_lightfield.save_fig_as_png('2D_shifts_overview')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:orange\">Task 2 - Implement a 2D-translation operation</span>\n",
    "\n",
    "Now we need to implement a function that translates an image given a 2D translation vector.\n",
    "\n",
    "Luckily, there are many python packages available that are doing this for you. You just have to find a good package (e.g. skimage or openCV (speed ! ) that will do this job for you)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can use the following code to test out if your translation method has worked well\n",
    "\n",
    "img = data[0,0]\n",
    "print(img.shape)\n",
    "\n",
    "plt.figure(figsize=(13,5))\n",
    "\n",
    "plt.subplot(141)\n",
    "\n",
    "tmp = lightfield.translate_image(img,50,0)\n",
    "print(tmp.shape) #Should be the same as img.shape\n",
    "plt.imshow(tmp) \n",
    "plt.title(\"Should shift to bottom\")\n",
    "\n",
    "plt.subplot(142)\n",
    "\n",
    "tmp = lightfield.translate_image(img,0,50)\n",
    "print(tmp.shape) #Should be the same as img.shape\n",
    "plt.imshow(tmp) \n",
    "plt.title(\"Should shift to right\")\n",
    "\n",
    "plt.subplot(143)\n",
    "\n",
    "tmp = lightfield.translate_image(img,75,75)\n",
    "print(tmp.shape) #Should be the same as img.shape\n",
    "plt.imshow(tmp) \n",
    "plt.title(\"Should shift to diagonally\")\n",
    "\n",
    "plt.subplot(144)\n",
    "\n",
    "tmp = lightfield.translate_image(img,-75,-75)\n",
    "print(tmp.shape) #Should be the same as img.shape\n",
    "plt.imshow(tmp) \n",
    "plt.title(\"Should shift to diagonally other direction\")\n",
    "\n",
    "util_lightfield.save_fig_as_png('translate_image_2d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:orange\">Problem: Aperture Simulation</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The light-field can be interpreted as capturing images with a much larger aperture. However, the light-field is 4-dimensional and you can interprete each view as a capture with a very small aperture.\n",
    "\n",
    "To remind yourself what the aperture is doing and what effect it is having on the captured images, please have a look at the following image. The linked article (where we've taken the image from is also quite instructive, if you want to give it another read)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![test](pictures/Aperture-Range.png)\n",
    "\n",
    "<br>\n",
    "\n",
    "from https://maratstepanoff.com/wp-content/uploads/2018/12/Aperture-Range.png\n",
    "Read more here: https://maratstepanoff.com/aperture-in-photography/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal will be to write an algorithm that allows to refocus the 4D-lightfield to any depth-position that we want. This depth will be given by the parameter $\\alpha$. \n",
    "\n",
    "However, to accurately estimate the refocusing we should use a circular aperture instead of a rectangular aperture.\n",
    "\n",
    "Why are we writing rectangular aperture here? If you remember how the light-field was captured, you'll realize that we captured this from a 17x17 grid which was rectangular.\n",
    "\n",
    "We will have to write an algorithm that takes this into account, so let's begin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:orange\">Task 1: Create the circular aperture</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we will create the circular mask. For this we have already implemented the function \"get_disk_aperture\" for you.\n",
    "\n",
    "Just run the following cell which is visualizing the circular aperture for different radii.\n",
    "\n",
    "\n",
    "HINT: Note that we are defining the circular aperture through its radius and not through the diameter. Because of this the shape of our aperture is always odd and we always ahve a well defined center."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,4))\n",
    "\n",
    "for k in range(8):\n",
    "    plt.subplot(2,4,k+1)\n",
    "\n",
    "    A_disk = util_lightfield.get_disk_aperture(k+1)\n",
    "    print(A_disk.shape)\n",
    "    im = plt.imshow(A_disk)\n",
    "    plt.title(\"R = \" + str(k+1))\n",
    "    code.add_colorbar(im)\n",
    "    \n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:orange\">Task 2: Task implement a pipeline for Compute Refocused</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's actually time to implement the refocusing algorithm. We will go through this step by step. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Subtask: Revisit call-by-reference in Python</span>\n",
    "\n",
    "Below is a small script that show-cases how call-by-reference works in Python. E.g. when we substract data from a larger images by assessing e.g. submatrices we're not doing copies of the data, but we're rather just getting the pointer to the location in memory.\n",
    "\n",
    "In Computational Photography we're often working with large data which is memory consuming. Our goal should be to write memory-efficient code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# An example of how call-by-reference in python works\n",
    "# You can read more here: https://ecco-v4-python-tutorial.readthedocs.io/ECCO_v4_Operating_on_Numpy_Arrays.html\n",
    "\n",
    "A = np.random.randn(10,10) # Create a random matrix\n",
    "\n",
    "plt.subplot(121)\n",
    "im = plt.imshow(A)\n",
    "plt.title(\"Original\")\n",
    "code.add_colorbar(im)\n",
    "\n",
    "# Define B to be a submatrix of A. This is call-by-reference and the entries of B and \n",
    "# respective entries in A share (!) the SAME memory (!)\n",
    "B = A[1:5,1:5]  \n",
    "print(B) \n",
    "B[:,:] = 1 # Reset all values inside to 1\n",
    "\n",
    "plt.subplot(122)\n",
    "im = plt.imshow(A) # Even though we only changed values in B, the values in A shold now be changed!\n",
    "plt.title(\"Call by Reference\")\n",
    "code.add_colorbar(im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 3: Write a image crop function</span>\n",
    "\n",
    "Now we'll need to write a function that crops part of the light-field image according to the size of the used aperture-mask.\n",
    "\n",
    "E.g. if we use a aperture-mask that has a diameter of 7 pixels we don't work on the complete lightfield but only with a sub-section of this data.\n",
    "\n",
    "Why are we doing this: If we work with only a region of 7x7 we only need to translate $7^2=49$ images instead of $17^2=289$. This saves a lot of computation time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get a mask for testing out\n",
    "A_mask = util_lightfield.get_disk_aperture(7)\n",
    "plt.imshow(A_mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# But should also work if no mask is provided (i.e. return the same data)\n",
    "data_cropped = lightfield.crop_part(data)\n",
    "\n",
    "print(\"If no mask is used\")\n",
    "print(data.shape)\n",
    "print(data_cropped.shape) # should be the same as data.shape since we're not using any mask\n",
    "\n",
    "# Should work for masks\n",
    "data_cropped = lightfield.crop_part(data,A_mask)\n",
    "\n",
    "print(\"If a mask is used\")\n",
    "print(data.shape)  # should be (17,, 17,  400, 700, 3) \n",
    "print(data_cropped.shape) # should be (13, 13, 400, 700, 3) for a radius of r = 7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just visualize that the crop has actually worked\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.imshow(data[:,:,200,550])\n",
    "plt.title(data[:,:,200,550].shape)\n",
    "plt.subplot(122)\n",
    "plt.imshow(data_cropped[:,:,200,550])\n",
    "plt.title(data_cropped[:,:,200,550].shape)\n",
    "\n",
    "util_lightfield.save_fig_as_png('cropped2d_example')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 4: Implement the shift_lightfield function</span>\n",
    "\n",
    "Now we have the following pieces already implemented:\n",
    "1. Get the shifts according to the refocusing formula\n",
    "2. Translate an image function\n",
    "3. Crop light-field to the correct size\n",
    "\n",
    "\n",
    "Now we need to shift each image in the light-field data according to corresponding shift that you have calculated above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We're recalculating how much we have to shift each images\n",
    "# This function should already be implemented by you\n",
    "DX, DY = lightfield.get_shifts(data_cropped.shape,alpha)\n",
    "\n",
    "# This is the function that you have to implement now\n",
    "tmp_lf = lightfield.shift_lightfield(data_cropped,DX,DY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test that the dimensions aftert shifting still match up:\n",
    "\n",
    "print(data_cropped.shape)\n",
    "print(tmp_lf.shape) # should be the same"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 5: Average the shifted lightfield</span>\n",
    "\n",
    "Now that you've shifted the images, all that is left to do is to average your shifted lightfield. If we wouldn't have an aperture mask this would be straightforward and we could just use the np.mean function.\n",
    "\n",
    "However, we have to account for the aperture mask. The easiest way to deal with this is to weigh the lightfield by multiplying the aperture function element-wise to the light-field data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If not mask is given just return the same matrix\n",
    "weighted_lf = lightfield.weight_shifted_ligthfield(tmp_lf)\n",
    "print(weighted_lf.shape)\n",
    "\n",
    "# If a mask is given weight the matrix!\n",
    "weighted_lf = lightfield.weight_shifted_ligthfield(tmp_lf,A_mask)\n",
    "print(weighted_lf.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Subtask: Check if the averaging function worked well</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just run the following cell. If you've implemented everything correctly, this should work out of the box."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.subplot(121)\n",
    "plt.imshow(weighted_lf[0,0,:,:]/255)\n",
    "plt.title(\"Should be 0\") # Why should this be 0 ?! Think about this\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.imshow(weighted_lf[5,5,:,:]/255)\n",
    "plt.title(\"Should be Full image\")\n",
    "\n",
    "plt.figure()\n",
    "# Choose a value that shows an interesting sup-aperture image\n",
    "my_x = 300\n",
    "my_y = 350\n",
    "plt.subplot(121)\n",
    "tmp = data[:,:,my_x,my_y,0]\n",
    "plt.imshow(tmp)\n",
    "plt.title(\"Original - \" + str(tmp.shape))\n",
    "plt.subplot(122)\n",
    "tmp = weighted_lf[:,:,my_x,my_y,0]\n",
    "plt.imshow(tmp)\n",
    "plt.title(\"Masked - \" + str(tmp.shape)) # Can you see the circle of the circular aperture mask ?\n",
    "\n",
    "util_lightfield.save_fig_as_png('2d_aperture_weighting')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:orange\">Task 6: Average the shifted lightfield</span>\n",
    "\n",
    "After weighting all that is left, is to average the image. \n",
    "\n",
    "NOTE: If there's no mask you can simply use np.mean to calculate the average. However, because there is an aperture mask you need to normalize your image correclty. The easiest way to do this is to simply calculate the sum over the light-field calcualte and then divide by the sum of the aperture_mask to calcualte the average. Make sure to cast your final result into uint8 so that you don't have any problems with image display in matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "refocused = lightfield.average_shifted_lightfield(tmp_lf,A_mask)\n",
    "plt.imshow(refocused)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:orange\">Task 7: Test your light-field implementation</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have combined the complete refocusing pipeline for you in the function \"compute_refocused\". Just run the following and see if it works.\n",
    "\n",
    "If it doesn't work, try to debug your functions inside until it works!\n",
    "\n",
    "In the cell after, we've implemented a small interactive widget that allows you to play around with your implementation more!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "radius = 9 # feel free to play around with this\n",
    "A_mask = util_lightfield.get_disk_aperture(radius) \n",
    "alpha = 0.5 # Feel free to play around with this\n",
    "refocused = util_lightfield.compute_refocused(data,alpha,A_mask)\n",
    "\n",
    "plt.figure(figsize=(15,10))\n",
    "plt.imshow(refocused)\n",
    "plt.title(\"Radius = \" + str(radius) + r\" | $\\alpha$ = \" + str(alpha))\n",
    "\n",
    "util_lightfield.save_fig_as_png('lightfield_refocused_2d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:orange\">Problem: Test your refocusing algorithm</span>\n",
    "\n",
    "Now that everything has been implemented, we want to test your refocusing algorithm in an interactive manner. Please use the ipywidget that we've implemented for you below.\n",
    "\n",
    "If all functions work nice "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(alpha,a,mask):\n",
    "    \n",
    "    A_mask = util_lightfield.get_disk_aperture(a)\n",
    "    print(A_mask.shape)\n",
    "    refocused = util_lightfield.compute_refocused(data,alpha,A_mask)\n",
    "    #if a == 0:\n",
    "    #    refocused = compute_refocused(data,alpha,A_mask)\n",
    "    #else:\n",
    "    #    refocused = compute_refocused(data[a:-a,a:-a],alpha)\n",
    "        \n",
    "    plt.figure(figsize=(15,10))\n",
    "\n",
    "    plt.imshow(refocused)\n",
    "    plt.title(r\"$\\alpha$ = \" + str(alpha) + \" | \" + \" D = \" + str(2*(a-1)+1) + \" | Mask = \" + str(mask))\n",
    "    plt.show()\n",
    "\n",
    "x_widget =  widgets.FloatSlider(value = 0.5,min=-1, max=3, step=0.1)\n",
    "a_widget =  widgets.IntSlider(value = 1,min=1, max=data.shape[0]/2+1, step=1)\n",
    "    \n",
    "interactive_plot = interactive(f, alpha = x_widget,a = a_widget, mask = True)\n",
    "output = interactive_plot.children[-1]\n",
    "output.layout.height = '15'\n",
    "interactive_plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}