How to work with non-cuboid layout prediction

[kazu0622]

No description provided.

[kazu0622]

Hello.
I tried some cuboid layout prediction and ended in success, but, failed in non-cuboid layout prediction. Layout-net failed estimating Boundary Map and Corner Map as if the input data are cuboid in my operation.

Could you give me some advice to work well for non-cuboid layout prediction and how to modify panoOptimization.m ?

[kazu0622]

Hello.
I attached an example failed predicting non cuboid layout.

I'd like to know how to get correct predictions for non-cuboid room images.

Thanks.

[zouchuhang]

@kazu0622 If you train with images of non-cuboid shape but with cuboid shape ground truth, your trained network will have cuboid predictions for all cases. I'd suggest you remove images with non-cuboid shape for training.
Otherwise, you can train with non-cuboid shape ground truth. I've provided training samples with a few non-cuboid ground truth (https://drive.google.com/drive/u/1/my-drive) and a pre-trained model (https://drive.google.com/file/d/1eqP8J1iFB3LriNxXMM2wGsP-AsMPf-6r/view?usp=sharing). You can play around with them. Remember not to train with the 3D parameter regressor since there're non-cuboid layouts now.

For optimizing non-cuboid room shape, you need to:

1. Determine whether to generate four, six or eight etc. walls by thresholding the score of the
   sixth strongest wall-wall boundary (e.g. 0.05 in our paper).
2. Note that there will be multiple solutions given non-cuboid layout when solving Eq. 2 in the paper. Enumerate all possible shapes (e.g. for room with six walls, there will be six variations) and choose the one with the best score in Eq. 2. I've provided sample optimization code (pano_line_solver_6.m) to solve for Eq. 2 in the /matlab/ folder.

[kazu0622]

Hello. Thank for your quick response!
The result I sent above was predicted by the pre-trained data panofull_joint_box_pretrained.t7 you've provided through github. Is it correct that non-cuboid prediction needs pre-trained data specialized for non-cuboid rooms? I'm going to try the pre-trained data you provided on above comment.

And thank you for sharing the information about "pano_line_solver_6.m". I've already tried this code, but it didn't work just replacing "pano_line_solver.m". Could you possibly tell us tip to work well?

Thank you.

[zouchuhang]

@kazu0622 If your trained data are all cuboid shape, then it's still workable to predict non-cuboid corners and edges.

You cannot just replace "pano_line_solver.m" with "pano_line_solver_6.m", since "pano_line_solver.m" assumes input of 8 corners and "pano_line_solver_6.m" assume input of 12 corners. You need to first initialize number of corners based on step 1 in #25 (comment)

[kazu0622]

Thank you for your advice. I'd like to have your comment a little more that why pre-trained data "panofull_joint_box_pretrained.t7" didn't work on a non-cuboid images.

Thank you.

[kazu0622]

@zouchuhang Thank you for all your comment for my questions.
You mentioned that trained data by all cuboid shape is workable to predict non-cuboid corners and edges.
Is the pre-trained data you gave as below
https://drive.google.com/file/d/1bg9ZP3_KA1kvTWpCh4wQ0PfAuCm4j0qa/view?usp=sharing
is also workable?

Thank you.

[zouchuhang]

@kazu0622 "panofull_joint_box_pretrained.t7" is trained with images of both cuboid and non-cuboid shape, but all with cuboid shape ground truth. Thus for inference, given non-cuboid shape, it will predict cuboid results just as it learns from the training data. Therefore one solution could be to remove the non-cuboid shape training samples and retrain the network, so that the network might focusing on learning to label edges and corners, regardless of the room shape. You can play around with this strategy, and see if it takes affect.

Otherwise, you can train with non-cuboid shape ground truth for non-cuboid sample. The pre-trained model I provide in #25 (comment) is trained with a mixture of cuboid shape (with cuboid gt) and non-cuboid shape (with non-cuboid gt) images, the network can thus be able to predict corners and edges for general Manhattan layout. You can play around with it.

[kazu0622]

@zouchuhang I tried pre-trained model for non-cuboid layout you gave and use panofull_joint_ab_pretrained.t7 to import the model, but failed. It seems that the parameter counts doesn't match between the networkmodel and the pre-trained model.

I'd like to know how to cope with this.

Thank you.

[zouchuhang]

@kazu0622 As mentioned in #25 (comment), you need to exclude the 3D parameter regressor in the model since there're non-cuboid layouts now.

[kazu0622]

@zouchuhang Thanks a lot! The non-cuboid prediction has worked. Before I try to import "pano_line_solver_6.m", I'm checking how it works. It seems that the manhattan line estimation work without processing both "mano_line_solver.m" and "pano_line_solver_6.m". But I've got an wrong room shape. I guess the roll of "pano_line_solver_6.m" is optimaizing the wrong shape. Is it natural I've got an wrong result without "pano_line_solver_6.m"?

[zouchuhang]

@kazu0622 The Manhattan Line estimation is a preprocessing step and is independent on the 3D reconstruction step. To better estimate room shape, please follow #25 (comment) "1. Determine whether to generate four, six or eight etc. walls by thresholding the score of the sixth strongest wall-wall boundary (e.g. 0.05 in our paper)."