In this paper, we present a new model-based video interpretation paradigm by unified modeling of static (bilateral, bidirectional) and temporal context. Contextual information is useful for video interpretation to enhance recognition rate and to reduce computational burden. Bilateral (spatial) context such as part relations and object relations facilitates object recognition within a frame (static image). Bidirectional context among place, objects and parts also reinforces visual interpretation by information exchange. Temporal context alleviates computational load in video interpretation using inter-frame information. Place label is estimated by an extended HMM by adding object context to conventional HMM. Static context provides bottom-up proposal to the multiple object tracking block. Experimental results in large scale indoor environment show the feasibility of the proposed video interpretation scheme.