We present an approach for detecting moving objects from a dynamic video sequence, using a stereo camera system. The detection of moving objects is a challenging problem, especially when backgrounds are also time varying due to the concurrent changes of moving objects and backgrounds. Most of the previous approaches have been limited to the use of appearance information such as colors and 2D motions. A Markov random field (MRF) approach based on geometric reconstruction is proposed to handle the concurrent motions in segmenting moving objects from dynamic backgrounds robustly. Our approach introduces a high-order likelihood to reduce the influence of mismatched features in the background. Our method also enables the consistent detection of moving objects across frames by enforcing an efficient temporal coherence term. In addition, we incorporate with a superpixel representation to avoid computational complexity. Experiments demonstrate the effectiveness of the proposed method.