Consumer devices having a stereo camera have become popular because of low-cost depth sensing capability. However, those systems usually suffer from low imaging quality and inaccurate depth acquisition in low-light conditions. To address the problem, we present a new stereo matching method with a color and monochrome camera pair. We focus on the fundamental trade-off that monochrome cameras have much better light-efficiency than color-filtered cameras. Our key ideas are compensating radiometric difference between two cross-spectral images and taking the full advantage of the complementary data. Consequently, our method produces both an accurate depth map and a high-quality image which are applicable to various depth-aware image processing. Our method is evaluated on various datasets and the performance of our depth estimation consistently outperforms state-of-the-art methods.