Spatio-Temporal Graph-RNN for Point Cloud Prediction


We introduce a Graph Neural Network, that given a point cloud sequence can make accurate prediction of future frames.

Point clouds do not have regular structure or explicit point to point correspondence across frames, this makes model dynamic point clouds a very challenging task.

To address this challenge we propose a Graph Neural Network that can process irregular point clouds sequences using graph structure. We design a Graph-RNN cell that can leverage learned features, describing the local topology, to form spatio-temporal graphs, from where temporal correlations can be extracted.

How far can we see into the future?

For example given 10 frames as input, we make the Graph-RNN predicts the next 10 frames.

Long-term prediction of MNIST dataset converted to Point cloud

We can see the a accurate prediction of the next 10 frames, this means: The network has memory.

For each point the Graph-RNN learn a state, describing the point behaviour. For a frame t the network, considers the states in the previous frame t-1. This means that the network learns point movements taking into consideration the previous movements of points, allowing the cell to retain temporal information. The states act as a memory retaining the history of movements and enabling for network to model long-term relationships over time