Home
Here you can find some additional information of things that are not mentioned in the front-page README.md.
Some example fitting results on the HELEN database, created with the fit-model example app:
 |
 |
 |
 |
 |
 |
To use the Basel Face Model 2009 in eos, you can convert it to the format that eos uses with the Python script in share/scripts/convert-bfm2009-to-eos.py. It will read the PublicMM1/01_MorphableModel.mat file from the BFM2009 distribution and save an eos .bin model.
The following files are required as well:
-
2D to 3D landmark correspondences. A basic version of some correspondences for ibug landmarks is provided in the file share/ibug_to_bfm2009.txt.
-
A file defining which vertices of the BFM2009 should be used as left and right contour in the contour fitting. A basic version is provided in share/bfm2009_model_contours.json, but it is not properly tested.
-
bfm2009_edge_topology.json. This file is actually taken from https://github.com/waps101/3DMM_edges and just converted to eos's json format.
To use the Basel Face Model 2017 in eos, there is also a Python script to convert it: share/scripts/convert-bfm2017-to-eos.py. It will read the model2017-1_bfm_nomouth.h5 or model2017-1_face12_nomouth.h5 file from the BFM2017 download and save an eos .bin model.
The script will convert the PCA expression model of the BFM2017 as well. To use the BFM2017 in eos, you currently need to use the devel branch of eos.
Note that the BFM2017 comes in two variants, model2017-1_bfm_nomouth and model2017-1_face12_nomouth, and they have a different mesh topology (i.e. the vertex indices for points on the face are different).
-
2D to 3D landmark correspondences. A basic version of some correspondences for ibug landmarks is provided in the file share/ibug_to_bfm2017-1_bfm_nomouth.txt. It's a starting point and may need some adjustments to get good fitting results. No file is provided for the
_face12_variant of the BFM, but it is easy to create - contributions are welcome. -
Todo: bfm2017-1_bfm_nomouth_model_contours.json. The vertices chosen may need adjustments to improve the quality of the contour fitting. No file is provided for the
_face12_variant of the BFM, but it is easy to create - contributions are welcome. -
bfm2017-1_bfm_nomouth_edge_topology.json or bfm2017-1_face12_nomouth_edge_topology.json, depending on the variant of the BFM that is used. These files are generated with the script share/scripts/generate-edgestruct.py.
In the fit-model app, the edge fitting is just used to fit the occluding face contour to the ibug contour landmarks. However, the edge fitting is very general, and can be used to fit to edges from an edge detector (e.g. Canny edges). In fact this is how it is used in the original paper.
To achieve this in eos, the fitting::find_occluding_edge_correspondences() function just has to be called with a list of edges, instead of the landmarks. For example like so:
Mat edge_image;
double canny_thresh = 150.0;
cv::Canny(image, edge_image, canny_thresh*0.4, canny_thresh);
vector<cv::Point> image_edges;
cv::findNonZero(edge_image, image_edges);
vector<Eigen::Vector2f> image_edges_; // Need to convert these points to Eigen::Vector2f
std::for_each(begin(image_edges), end(image_edges), [&image_edges_](auto&& p) { image_edges_.push_back({ p.x, p.y }); });
auto mesh = morphable_model.get_mean();
auto edge_correspondences = fitting::find_occluding_edge_correspondences(mesh, edge_topology, rendering_params, image_edges_);
This will return a list of edge correspondences that can then be added to the landmark correspondences in the subsequent fitting.