In this project, the goal is to segment rover, background, and shadow (if any). In this part of the report we are going to use simple linear iterative clustering (SLIC), and adaptive SLIC for initial segmentation. Next, we want to implement a region adjacency graph (RAG) on the regions obtained from the segmentation algorithm. Finally, to handle the over-segmentation, Hierarchical merging, and Normalized cuts will be used. The illustration of the whole procedure is shown in Figure 1. In the next sections, implementation, comparison, and interpretation of the models will be discussed in detail.
Our data consists of four images. All of them are in RGB color space. Picture 0 has three objects including rover, background, and shadow; and the rest of them are without shadow. Figures 1 to 4 are the original images before any processing.
Figure 1 and 2
Figure 3 and 4
Figure 5
Simple linear iterative clustering (SLIC) utilized a k-means algorithm to generate superpixels that are supposed to adherence object boundaries in the image. It has two parameters to adjust which are the number of superpixels to generate and compactness. Moreover, the input image should be in CIELAB color space which we utilized skimage.segmentation.slic function that automatically changes the color space of the input image. To begin with, we set 200 for the k (number of superpixels), and 10 for compactness which is the default of their method 1. Usually, compact superpixels are more desirable but increasing the compactness may result in reduced boundary adherence [1]. Figure 5 shows the output label of the SLIC algorithm, and Figure 6 shows superpixels with boundaries in the original image. At this stage, we should tune these two parameters in a way that superpixels boundaries do not exceed the boundaries of the objects which in Figure 6 is not satisfied at certain areas circled by a red line. Therefore, in the next attempt, we can make it more compact in order to approach the desired over-segmentation. In Figure 7, it is shown that by increasing the compactness the result is more desirable yet some superpixels do not adherent the boundaries which are shown in a red circle. This problem should be because of the number of segments. Since the algorithm is supposed to generate superpixels roughly equal in size. Therefore, to continue tuning, we can increase the number of segments to generate more desirable superpixels. Figure 8 shows the results of increasing the number of segments to 600. The mentioned spot is segmented a little better by this tuning yet it can be better.
Figure 6 and 7
Figure 8
Furthermore, in the method 1 they stated that the number of iterations is 10 which is suitable for most images, but we intend to investigate this further. To do so, we increased the number of iterations to 50. The result is shown in Figure 9. Also, Figure 10 shows the above-mentioned spot in Figure 9 which is segmented very well. Consequently, we select these parameters as a well-tuned one, although in some other cases are still not perfect. Figure 11 shows the over-segmentation result obtained by the SLIC algorithm after parameter tuning. SLIC segmentation on the original image and without that for picture 2 are shown in Figures 12 and 13 respectively. Moreover, these results for pictures 1 and 0 are shown in Figures 14, 15, 16, and 17 respectively. By looking at the results of SLIC in other pictures, we can conclude that the parameters are promising, and can help the merging algorithms (Hierarchical and Ncuts) very well. In the next part, we are going to try adaptive SLIC and compare it with SLIC.
Figure 9 and 10
Figure 11 and 12
Figure 13 and 14
Figure 15 and 16
Figure 17
Adaptive SLIC is similar to the previous version of it except it automatically adapts the compactness based on the region. Therefore, there is no need to tune the compactness and all we have to do is define the number of segments. In order to do a fair comparison between the SLIC methods, we assign 600 for the number of segments (superpixels). Figure 18 shows the segmentation boundaries on the original image and also Figure 19 shows the segmentation labels with the mean intensity of each superpixel. It can be seen that in challenging spots, the accuracy of segmentation is not very well and some superpixels do not adhere to the boundaries of the rover. Figure 20 shows the mentioned spot obtained from SLIC and an adaptive version of it.
Figure 18 and 19
Figure 20 The left picture is obtained from the Adaptive SLIC and the right obtained from SLIC after parameter tuning
As can be seen in Figure 20, some superpixels (shown by the red line) exceed the rover’s boundaries. To conclude, an adaptive version of SLIC cannot overcome the refined version of SLIC. In the next Figures (21 to 26) the segmentation result and labels of pictures 2, 1, and 0 are shown respectively.
Figure 21 and 22
Figure 23 and 24
Figure 25 and 26
Region Adjacency Graph is a graph that connects the superpixels and assigns a weight for each connection. This weight represents the Euclidean distance between the average of two superpixels. The more the weight, the more different the mean of those adjacent regions. Figure 27 shows the initial RAG of picture 3 obtained from the SLIC algorithm. Higher values of difference in the mean color of regions are shown with the lighter color. Therefore, the boundaries weights are more than 150 roughly, and regions similar together in terms of the mean of superpixels, have weights less than 25 roughly. In the next Figures (28 to 33), the initial RAG of pictures 2, 1, and 0 are shown respectively.
Figure 27 and 28
Figure 29 and 30
This method merges the regions with a weight less than a threshold. Therefore, based on the color bar of the previous figures we can choose a proper value for the threshold. In this section, we start investigating from pictures 0 to 3. To begin with, based on the color bar we assign the threshold to 40. The new RAG is shown in Figure 31 with new weights. Also, the segmentation result is shown in Figure 32. As it can be seen, the overall performance is very well, but in some regions which are close in color (weight is low), the segments do not adhere to the boundaries. For instance, the edge of the rover’s scope is not well segmented which is circled by the red line in Figure 32. Perhaps, by reducing the threshold we can overcome this issue a little.
Figure 31 and 32
Let assign 25 for the threshold. Figure 33 shows the result for picture 0. The edge of the scope is better segmented but on the other hand, we faced over-segmentation for the background which is not desirable. Furthermore, for other pictures, based on their color bar we choose a proper threshold in a way that adheres to the boundaries as much as possible. After trying several values for the threshold for picture 1, the best result is shown in Figure 34. It adheres to the boundaries of the rover, although the background is not merged to a unit segment. Nevertheless, as long as it does not exceed the boundaries of the rover, it is acceptable and we can apply further algorithms like morphological operations to merge the background into a unit region. For the next picture, we initially assign 170 based on the color bar and then gradually change it to a better value. After some iterations, we assigned 110 for picture 2. Figure 35 shows the segmentation result of this picture. For the last picture, by trying a couple of values the best result is obtained with a threshold equal to 110. Nonetheless, in some areas, the segments exceed the boundaries of the object which are constant even for thresholds as low as 45. This is because of the similarity between the mean values of the mark on the rover and the background. Other remaining small superpixels again can be merged by morphological operations which we did in the previous project of the course.
Figure 33 and 34
Figure 35 and 36
Consequently, in order to prevent repetitive over-segmentation, we let the threshold be high. In this way, we can deal with the small remaining regions easier. Moreover, we tried the SLIC over-segmentation with a much less number of segments which resulted in better outcomes. Figure 37 is the final result of picture 0 after hierarchical merging which is exactly three segments as we expected although the edge of the scope is not well segmented. This is the weakness of the method and to the best of our efforts, it is the best possible result. Furthermore, the final segmentation of pictures 1, 2, and 3 are shown in Figures 38 to 42.
Figure 37 and 38
Figure 39 and 40
Figure 41 and 42
The normalized cuts algorithm is based on a generalized eigenvalue problem that calculates the similarity and dissimilarity between the groups of the graphs. This method has two parameters to tune that is threshold and number of cuts. The threshold defines the margin to control the dividing of the subgraphs. Moreover, the number of cuts is just to initialize the algorithm, and then the algorithm itself can investigate further to find the optimal value for it. By investigating the values for the threshold finally, we assigned 0.005 for it and 30 for the initial number of cuts. Figure 43 shows the result of this configuration in picture 0. Again, the segmentation is suffering from crossing boundaries in superpixels similar together in terms of color (circled by the red line). This is the weakness of both hierarchical and NCuts methods. Although the quality of segmentation is very well in other areas, after trying several numbers of thresholds for the algorithm, it is not able to merge regions of the rover, so the rover is considered as two superpixels which are not proper. This has happened to picture 3 which the final segmentation result has 4 segments although it is expected to be 2. The final segmentation result of picture 3 is shown in Figure 47. On the other hand, pictures 1 and 2 can merge properly with the low crossing of boundaries. Figures 44 and 45 show the result of segmentation of picture 1 after NCuts merging which is crossing the boundaries of the rover (circled by the red line).
Figure 43 and 44
Figure 45 and 46
Figure 47
To conclude, we can say that in most cases small numbers of segments in the SLIC algorithm can help the merging algorithms better. This is because of the smaller the superpixels, the more variance the mean of the superpixels. Therefore, large numbers of segments make the merging hard and lead them to more segments even after merging. Consequently, we prefer a small number of segments and the average value for compactness. Moreover, between SLIC and Adaptive version of SLIC, the results in the Adaptive SLIC section, shows that with the same number of segments, SLIC overcomes the adaptive version in terms of not crossing the boundaries. This has been discussed in detail in the Adaptive SLIC section. Besides, the Adaptive version does not work well with the low number of segments and crosses the boundaries frequently. For the merging algorithm, experiments show that hierarchical merging works better since the final number of segments is as it is expected, but on some occasions, Ncuts fails to merge to the proper number of segments. Nevertheless, Ncuts is better at adherence to the boundaries. To conclude, each of the algorithms has its pros and cons, and based on the application and our priority we can choose each of them or use state-of-the-art algorithms.