Structure from Motion (SfM) is a pipeline that allows three-dimensional reconstruction starting from a collection of images. How to choose the SfM pipeline best suited for a given task? We report a comparison of different state-of-the-art SfM pipelines in terms of their ability to reconstruct different scenes. We also propose an evaluation procedure that stresses the SfM pipelines using real dataset acquired with high-end devices as well as realistic synthetic dataset. To this end, we created a plug-in module for the Blender software to support the creation of synthetic datasets and the evaluation of the SfM pipeline. The use of synthetic data allows us to easily have arbitrarily large and diverse datasets with, in theory, infinitely precise ground truth. Our evaluation procedure considers both the reconstruction errors as well as the estimation errors of the camera poses used in the reconstruction.

Here we made publicly available the synthetic datasets that we created for the evaluation of structure for motion pipelines.

DATASET IVL-SYNTHSFM

The dataset used in our paper is shown in the figure below. Models a-e are synthetic models. Model f  is a real statue.

Click here to download the dataset : 

  IVL-SYNTHSFM.zip (298.6 MiB, 889 hits)

If you use IVL-SYNTHSFM, please cite the following paper:

@article{bianco-sfm2018,
 author = {Bianco, Simone and Ciocca, Gianluigi and Marelli, Davide},
 year = {2018},
 title = {Evaluating the Performance of Structure from Motion Pipelines},
 volume = {4},
 number = {8},
 journal = {Journal of Imaging},
 doi = {10.3390/jimaging4080098}
}

DATASET IVL-SYNTHSFM-v2

This dataset is an extension of the IVL-SYNTHSFM. It contains the same models but the photos have been generated in different ways. Each model is placed in a reference scene and is rendered under different lighting and camera conditions. For each model, 8 scenes are created and for each scene 100 images are taken from different points of view. The dataset is thus composed of 8 sets as follows:

The dataset can be downloaded from this Mendeley Data link: Download IVL-SYNTHSFM-v2

Details on data and acquisition setup can be found in the article: IVL-SYNTHSFM-v2: A synthetic dataset with exact ground truth for the evaluation of 3D reconstruction pipelines

If you use IVL-SYNTHSFM-v2, please cite the following paper:

@article{marelli2020IVLSYNTHSFMv2,
 author = {Marelli, Davide and Bianco, Simone and Ciocca, Gianluigi},
 year = {2020},
 pages = {105041},
 title = {IVL-SYNTHSFM-v2: A synthetic dataset with exact ground truth 
          for the evaluation of 3D reconstruction pipelines},
 journal = {Data in Brief},
 doi = {10.1016/j.dib.2019.105041},
 issn = {2352-3409}}

SOFTWARE SFM FLOW (Blender Add-on)

SfM Flow is a comprehensive toolset for the evaluation of 3D reconstruction pipelines. It provides tools for the creation of datasets composed of synthetic images, the execution of 3D reconstruction Structure from Motion pipelines, and the evaluation of the obtained results. SfM Flow integrates all these features in the popular modeling and rendering software Blender.
The IVL-SYNTHSFM-v2 dataset has been generated using SfM Flow, while the original IVL-SYNTHSFM used a preliminary version of the add-on.

Read more about SfM Flow in the paper SfM Flow: A comprehensive toolset for the evaluation of 3D reconstruction pipelines.

Get the SfM Flow Blender add-on from GitHub.

If you use SfM Flow, please cite the following paper:

@article{marelli2022sfmflow,
 author = {Marelli, Davide and Bianco, Simone and Ciocca Gianluigi},
 year = {2022},
 pages = {100931},
 title = {SfM Flow: A comprehensive toolset for the evaluation of 3D reconstruction pipelines},
 volume = {17},
 journal = {SoftwareX},
 doi = {10.1016/j.softx.2021.100931},
 issn = {2352-7110}}

@article{marelli2022sfmflow,
 author = {Marelli, Davide and Bianco, Simone and Ciocca, Gianluigi},
 year = {2022},
 pages = {100931},
 title = {SfM Flow: A comprehensive toolset for the evaluation of 3D reconstruction pipelines},
 volume = {17},
 journal = {SoftwareX},
 pdf = {https://www.sciencedirect.com/science/article/pii/S2352711021001692},
 doi = {10.1016/j.softx.2021.100931},
 issn = {2352-7110},
 projectref = {http://www.ivl.disco.unimib.it/activities/evaluating-the-performance-of-structure-from-motion-pipelines/}}

@article{marelli2020IVLSYNTHSFMv2,
 author = {Marelli, Davide and Bianco, Simone and Ciocca, Gianluigi},
 year = {2020},
 pages = {105041},
 title = {IVL-SYNTHSFM-v2: A synthetic dataset with exact ground truth for the evaluation of 3D reconstruction pipelines},
 journal = {Data in Brief},
 pdf = {/download/IVL-SYNTHSFM-v2.pdf},
 doi = {10.1016/j.dib.2019.105041},
 issn = {2352-3409}}

@article{bianco-sfm2018,
 author = {Bianco, Simone and Ciocca, Gianluigi and Marelli, Davide},
 year = {2018},
 title = {Evaluating the Performance of Structure from Motion Pipelines},
 volume = {4},
 number = {8},
 journal = {Journal of Imaging},
 pdf = {/download/bianco-sfm2018.pdf},
 doi = {10.3390/jimaging4080098},
 projectref = {http://www.ivl.disco.unimib.it/activities/evaluating-the-performance-of-structure-from-motion-pipelines/}}

@inproceedings{bianco2018blender-plugin,
 author = {Marelli, Davide and Bianco, Simone and Celona, Luigi and Ciocca, Gianluigi},
 year = {2018},
 pages = {1-5},
 title = {A Blender plug-in for comparing Structure from Motion pipelines},
 organization = {IEEE},
 booktitle = {2018 IEEE 8th International Conference on Consumer Electronics - Berlin (ICCE-Berlin)},
 doi = {10.1109/ICCE-Berlin.2018.8576196},
 issn = {2166-6822}}