Our video prediction approach uses sparse motion field which combines deep RL technique to discover the critical sparse positions and their motion vectors. We apply POBMC method for final frame synthesis. Our prediction shows similar PSNR and SSIM score compared with SOTA but much better human vision perceptual LPIPS score.

Our model combines video compression POBMC technique and modern DL and RL. Unlike direct generation, our model explicitly models the motion tendencies. Different from dense method which requires artificial constraint, ours preserves the true motions. Besides, our model size is small and we only need fewer data to achieve good generalization.

From industry application perspective, video prediction can be used in any application that needs to predict the future, such as smart autonomous vehicles. Or video prediction can benefit other tasks by predicting the future first such as activities predictions. This method can also be applied on video compression. We only need to transmit a small amount of motion vectors and their location information.