This paper deals with a problem of extracting moving objects without fixing a camera. To simplify the problem, we use a panoramic background from a panning camera and generalize standard background subtraction methods to work with non-static cameras. The technical steps are two-fold. While we collect input images to establish a panoramic background model within the panning range, any input images containing moving objects are classified. Then, the moving objects are extracted using a modified manner of statistical inference and template matching. The proposed video surveillance system is also evaluated with general indoor and outdoor datasets captured by our implemented system.