Most vision-based UAV (Unmanned Aerial Vehicle) navigation algorithms extract manmade features such as buildings or roads, which are well structured in urban terrain, using the CCD camera. But in the mountain area, extracting, matching or tracking features is more difficult than doing the tasks in the urban terrain. And a CCD camera cannot carry out the computer vision algorithm that is required for the UAV navigation in the night or under dark situation. In this paper, we introduce a new approach for vision based UAV localization method. In our proposed system an UAV uses only DEM (Digital Elevation Map), IR (Infra-Red) image sequences and an altimeter. It can estimate its position and orientation with hypothesis and verification paradigm.