1. Multi-Scale Models
All the methods implemented in
menpofit are built in a multi-scale manner, i.e. in multiple resolutions.
In all our core classes, this is controlled using the following three parameters:
First, the size of the training images is normalized by rescaling them so that the scale of their ground truth shapes matches the scale of this reference shape. In case no reference shape is provided by the user, then the mean of the ground shapes is used. This step is essential in order to ensure consistency among the extracted features of the images.
This parameter is used to rescale the reference shape so that the diagonal of its bounding box matches the provided value. This rescaling takes place before normalizing the training images' size. Thus, this parameter controls the resolution of the model at the highest scale.
A tuple with the scale value at each level, provided in ascending order, i.e. from lowest to highest scale. These values are proportional to the final resolution achieved through the reference shape normalization.
The multi-scale training pattern followed by all
menpofit methods is summarized in the following pseudocode:
REQUIRE: images, shapes PARAMETERS: reference_shape, diagonal, scales 1. IF reference_shape IS None: 2. reference_shape = mean_pointcloud(shapes) 3. reference_shape = reference_shape.rescale_to_diagonal(diagonal) 4. scaled_images, scaled_shapes = rescale_to_reference_shape(images, shapes, 5. reference_shape) 6. FOR scale IN scales: 7. scaled_images, scaled_shapes = rescale(images, shapes, scale) 8. scale_model = train_model(scaled_images, scaled_shapes)
2. Holistic vs. Patch Features
The methods implemented in
menpofit can be roughly separated in two categories, based on the appearance representation approach:
- Holistic Models: These are models that use a holistic appearance, which means that the whole texture that lies within the deformable object is taken into account.
- Patch-based Models: These are models that employ a patch-based appearance representation, which means that rectangular patches are extracted that are centered around each landmark point.
Both hositic and patch-based methods have a holistic_features argument which expects the
callable that will
be used for extracting features from the training images. The patch-based methods build on top of the computed
holistic features in order to extract patches and can have two behaviours depending on their nature:
- Those that simply extract patches from the holistic_features representation of the training images, given a patch_shape.
- Those that extract patches from the holistic_features representation of the training images, given a patch_shape, and additionally compute some patch_features.
3. Incremental Training
Most deformable models that exist in
menpofit can be trained in an incremental fashion. This can be very useful
in case you want to train a very powerful model on a large amount of training images, which would be impossible to
do at once due to memory constraints. The incremental training can be performed by using the batch_size
parameter, which will split the provided set of training images into batches of the specified size.
4. Training Example
Given the above basic assumptions and using the
training_images loaded in the Importing Images section,
an example of a typical call for training a deformable model using
from menpofit.aam import HolisticAAM from menpo.feature import igo aam = HolisticAAM(training_images, reference_shape=None, diagonal=150, scales=(0.5, 1.0), holistic_features=igo, verbose=True)
verbose flag allows printing the training progress:
- Computing reference shape Computing batch 0 - Building models - Scale 0: Done - Scale 1: Scaling images: [= ] 14% (119/811) - 00:00:01 remaining
Information about any trained model can be retrieved by:
Holistic Active Appearance Model - Images scaled to diagonal: 150.00 - Images warped with DifferentiablePiecewiseAffine transform - Scales: [0.5, 1.0] - Scale 0.5 - Holistic feature: igo - Appearance model class: PCAModel - 810 appearance components - Shape model class: OrthoPDM - 132 shape components - 4 similarity transform parameters - Scale 1.0 - Holistic feature: igo - Appearance model class: PCAModel - 810 appearance components - Shape model class: OrthoPDM - 132 shape components - 4 similarity transform parameters