fit-model-simple gets Segmentation fault when using BFM2017 model in Ubuntu 16

[kidapu]

When I run this command, I met Segmentation fault.

$ ./fit-model-simple \
    -m ../../share/bfm2017-1_bfm_nomouth.bin \
     -p ../../share/ibug_to_bfm2017-1_bfm_nomouth.txt \
     -i ../../examples/data/image_0010.png \
     -l ../../examples/data/image_0010.pts
Segmentation fault (core dumped)

When I analyze core file with gdb, the result is below.

$ gdb fit-model-simple core.fit-model-simpl.15555
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from fit-model-simple...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 15555]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./fit-model-simple -m ../../share/bfm2017-1_bfm_nomouth.bin -p ../../share/ibug'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000419014 in eos::render::render_affine(eos::core::Mesh const&, Eigen::Matrix<float, 3, 4, 0, 3, 4>, int, int, bool) ()
(gdb)
(gdb)
(gdb)

The fit-model-simple run well with using sfm model. But when using BFM model, it gets seg fault.

./fit-model-simple \
	-m ../../share/sfm_shape_3448.bin \
	-p ../../share/ibug_to_sfm.txt \
	-i ../../examples/data/image_0010.png \
	-l ../../examples/data/image_0010.pts

I generated bfm2017-1_bfm_nomouth.bin from model2017-1_bfm_nomouth.h5, is downloaed from this url.

python ./share/scripts/convert-bfm2017-to-eos.py 

Please let me know if you have any solution.

[patrikhuber]

Hi!

We'll need to know on what line it crashes, and ideally you'd have a look at the call stack and the variables. I recommend you do a RelWithDebugInfo build and then when it crashes, use a graphical debugger to check. On Windows I'd use Visual Studio, on Linux you could use CLion or perhaps QtCreator.

[kidapu]

@patrikhuber
Hi! I captured call stack of QtCreator.

I think this error happens because the size of mesh.texcoords == 0 in render_affine.hpp L102.

eos/include/eos/render/render_affine.hpp

Line 104 in 9a51161

glm::tvec2<float>(mesh.texcoords[i][0], mesh.texcoords[i][1])});

[patrikhuber]

Yes indeed. The BFM doesn't come with texture coordinates, and you're trying to extract/remap texture, so that for sure can't work.
If you need texture remapping, you have to get or create some uv coords for the BFM. The converter scripts in eos have a note on how you can conveniently add them.

[patrikhuber]

I agree there should perhaps be a better error message than just a segfault, so we can leave this open until we've added better error messaging.

[kidapu]

@patrikhuber
Mmm...

I specified texture_coordinates, face texture_triangle_indices in convert-bfm2017-to-eos.py, and generated new bfm2017-1_bfm_nomouth.bin , and re-run ./fit-model-simple.

But I met same seg fault error, it's because of mesh.texcoords is 0.

The uv params I specified is following.

1) texture_coordinates

https://drive.google.com/open?id=19LfglO8Fjh0wEU8mSO2yeFemvTeiKZcP

[[  5.   5.]
 [  6.   5.]
 [  7.   5.]
 ...
 [145. 201.]
 [146. 201.]
 [147. 201.]]

2) texture_triangle_indices

https://drive.google.com/open?id=1YguZtPUA1SHS87pt67SR_S7CRoeoiXYH

[[    1   247   246]
 [    1     2   247]
 [    2   248   247]
 ...
 [43804 43805 43865]
 [43805 43866 43865]
 [43805 43806 43866]]

[patrikhuber]

Uv coordinates are usually between 0 and 1, and this is what eos expects. The values of your texture_coordinates look a bit weird. Also you better check if the texture_triangle_indices should start at zero or one, this should be in the documentation, if not, you can check the ones of the SFM.

But if you load a model with texture coords (even if they're wrong), mesh.texcoords.size() shouldn't be zero. So you have to check why, when you're adding texcoords, it's still empty.
Save the model in the BFM converter script in Python and then load the model again in the Python prompt, and check that MorphableModel::get_texture_coordinates() returns something.
It should be straightforward.

[kidapu]

@patrikhuber
After I updated uv coordinated between 0-1, MorphableModel::get_texture_coordinates() resturns the value. But I met another error when I run ./fit-model-simple.

I think my uv texture indices is lower than vertex indices... Sorry...
I should create good uv texture indices. ( But i don't have idea now...)

I have have another question. Is there a way to run ./fit-model-simple without remapping texture?

[patrikhuber]

You can just comment the line in the fit-model-simple code that extracts the texture. The example is well documented and if you have a look at it, you'll find it immediately. Then it should of course work just fine.

In eos v1.1.0, I've done some improvements with regards to meshes that have a different number of texture triangle indices to the mesh's triangle indices. The eos::render::v2::extract_texture(...) might actually work, but is still experimental. There's a bit more info in the v1.1.0 release notes.

[saikiran321]

Use this texture coordinates from 3DMMasSTN repository for BFM model.

[kidapu]

@saikiran321
Thanks. Do you know how to generate texture_triangle_indices ? It's compatible with texture coordinates in 3DMMasSTN repository.

[kidapu]

@patrikhuber
I commented out the code around texture like this.
https://gist.github.com/kidapu/fb0be31b5eeaa39dcb7a2c276ee5dbba#file-fit-model-simple-cpp-L182-L191

The generated obj is below. It seems that not only the texture, but also the expression is not much reflected in BFM. Is this right?

On the other hands, mouth is opened when I tried to use same picture and SFM model.

[patrikhuber]

@kidapu The fit-model-simple example does not do colour fitting. Please make sure you understand what you're running, our paper (linked in the readme) would be a good starting point. What you're seeing is the BFM's mean colour, that's all.
As for the expressions, you probably need to change the regularisation value (lambda), the BFM expression model would likely have a different good value than the SFM.

[kidapu]

@patrikhuber
Sorry, My english is not good. I already knew fit-model-simple doesn't do color fitting. I want to fit expressions with BFM.

Although I changed the regularisation value(lamda) between 0.005 - 10.0,
The 3D model of BFM does not open its mouth.

※I changed the value of lamda here.

I think the accuracy of BFM is lower than that of SFM.

[patrikhuber]

@kidapu Make sure you only change the lambda of the expression fitting, and not the lambda of both the identity and expression fitting. It looks from your screenshot like you might only have changed lambda for the identity shape fitting.
Well, it might likely be needed to change both for the BFM, but good regularisation values for identity and expression will likely be different values, and not the same value.

[kidapu]

@patrikhuber
I think lambda in fit-model-simple seems to be used in only one place.

I think It's here, is calling fitting::fit_shape_to_landmarks_linear.

Is there any other place where I can specify lamda for expression fitting?
Or do I need to update the fit-model-simple for BFM?

(I'm sorry I ask questions one after another ... 🙇)

[patrikhuber]

@kidapu Oh okay I see what's going on. fit-model-simple does not fit expressions at all actually. As you correctly posted, it calls fitting::fit_shape_to_landmarks_linear(...) and that fits only the (identity) shape. You can easily tell by the fact that that function only returns one vector of coefficients.
Actually the README documents this too, it says:

If you are just getting started, it is recommended to have a look at fit-model-simple too, as it requires much fewer input, and only fits pose and shape, without any blendshapes or edge-fitting.

What you probably want is to use fit_shape_and_pose(...), which fits pose, shape and expressions in an iterative way (and has a regularisation parameter for both identity and expressions). (Or if you don't want the additional complexity of contour fitting, just have a look at how that function works and use it without contour fitting).

[kidapu]

@patrikhuber
Although I try to use fit_shape_and_pose(...), I don't know how to get the argument of BFM's edge_topology. Please let me know if you know any hints m(_ _)m

[patrikhuber]

@kidapu As I said you can modify that function to not use contour fitting if you don't want to do contour fitting. If you do, there's also instructions in the eos wiki of how to get or generate these files, and all the scripts are in the share/scripts/ directory of eos.

[kidapu]

@patrikhuber
I keep trying to use fit_shape_and_pose(...), I have read eos wiki, and I have been able to download some files. And I think fit-model.cpp is good reference for me to use fit_shape_and_pose(...).

When I run fit-model.cpp with BFM, I got following error.

I think this happens becsuse share/expression_blendshapes_3448.bin is not compatible with BFM. Do you know how to get expression_blendshape for BFM?

[patrikhuber]

@kidapu Yes, of course you can't combine the blendshapes from the SFM with the BFM, as they would have a completely different topology/number of vertices.

If you're using the BFM2009, then I believe there's an expression model "out there on the internet", made from the FaceWarehouse data, available for academic use I think. I think it's originally from the 3DDFA paper and the one we also used for our 3DMMasSTN work, where you can find a link. People have also discussed this already in other eos issues, try searching through them.

If you're using the BFM2017, it already includes an expression PCA model, so you can skip loading any separate blendshapes.

[BenjBarral]

Use this texture coordinates from 3DMMasSTN repository for BFM model.

Hi,

I am trying to perform texture extraction from BFM fitting.
Thank you for the link to the UV coordinates, but it seems to be inadequate with the BFM2017 :
the BFM 2017 model has 53149 vertices, whereas the BFM_UV.mat has 53490 vertices.

Have you by any chance tried texture extraction with BFM, and could therefore refer to texture coordinate data for it ?

Thank you

[patrikhuber]

Hi @BenjBarral, I don't have any uv coords for the BFM2017. Perhaps you can retopologise the ones from the BFM2009, ask the authors, create them by yourself, search the web/publications if anyone has a set of uv coords for the BFM2017, etc.

[patrikhuber]

I am closing this, since the original issue was a segfault in extract_texture(...) when handing it a model that doesn't have texture coordinates - which obviously can't work, but I agree a better error handling would be good. There is already #228 open that addresses that.

Feel free to still use this issue if people want to continue the discussion about BFM2017 texture coordinates, I guess if somebody had/created/shared them, it would be useful to quite a few people.

[lvZic]

@patrikhuber I keep trying to use fit_shape_and_pose(...), I have read eos wiki, and I have been able to download some files. And I think fit-model.cpp is good reference for me to use fit_shape_and_pose(...).

When I run fit-model.cpp with BFM, I got following error.

I think this happens becsuse share/expression_blendshapes_3448.bin is not compatible with BFM. Do you know how to get expression_blendshape for BFM?

hi, have u fixed it and generating goog result successfully, i met same problem using bfm 2009 model.

[patrikhuber]

Hi @lvZic, Indeed share/expression_blendshapes_3448.bin are expression blendshapes for the SFM_3448 and logically won't be useful for any other models. If you want to use the BFM with expressions, you have to find (or create) an expression model for it. I think there exists an expression model for the BFM created from FaceWarehouse. If you search through eos's closed issues, you should find out more infos about it, I've pointed other people into that direction before. Also it might be worth checking the latest BFM version, I think they might now include a limited set of expressions.