We present a robust visual lock-on framework for an unmanned aerial vehicle (UAV) that utilizes geometric relations between the UAV pose and the 3D local map defined by the positions of the target and natural landmarks. Experimental results using real datasets demonstrate the robustness of the proposed method compared to the-state-of-the-art visual tracking methods.