For simplicity, we evaluate FG-ARI on all frames of the video (incl. All SAVi and SAVi++ models reported above use bounding boxes of the first video frame as conditioning signal. ModelĪll results are in terms of FG-ARI (in %) on validation splits.

We provide updated results for our released configs and the MOVi datasets with version 1.0.0 below. MOVi-A is approximately comparable to the 'MOVi' dataset used in our ICLR 2022 paper, whereas MOVi-C is approximately comparable to 'MOVi++'. The released MOVi datasets as part of Kubric differ slightly from the ones used in our ICLR 2022 paper and are of slightly higher complexity (e.g., more variation in backgrounds), results are therefore not directly comparable. SAVi++ is able to better handle real world videos with more complexities such as camera movements and complex object shapes and textures. SAVi++ also adds data augmentation and training on depth targets. SAVi++ uses a more powerful encoder than SAVi-L that adds transformer blocks to the ResNet34. This repository contains also the SAVi++ model configurations from our NeurIPS 2022 paper. SAVi-L is similar to SAVi-M except that it uses larger ResNet34 encoder and slot embedding. SAVi-S is trained and evaluated on downscaled 64圆4 frames, whereas SAVi-M uses 128x128 frames and a larger CNN backbone. We here refer to these models as SAVi-S, SAVi-M, and SAVi-L. This repository contains the SAVi model configurations from our ICLR 2022 paper.

To run SAVi or SAVi++ on other MOVi dataset variants, follow the instructions above while replacing movi_a with, e.g. You can also copy it into a different location and set the data_dir accordingly. In order to use the local copy simply set data_dir = './' in the config file configs/movi/savi_conditional_small.py. The resulting directory structure will be as follows.

Slot video