We present an efficient technique to optimize color consistency of a collection of images depicting a common scene. Our method first recovers sparse pixel correspondences in the input images to align local image patches. The corresponding pixel intensities from these patches are stacked into a matrix with many missing entries. We show that this matrix satisfies a rank two constraint under a simple color correction model. We present a robust low-rank matrix factorization method to estimate the unknown parameters of this model. These parameters can be viewed as pseudo white balance and gamma correction parameters for each input image. Using them, we improve color consistency of the input images or perform color transfer with any input image as the source. Our approach is insensitive to outliers in the pixel correspondences thereby precluding the need for complex pre-processing steps. We demonstrate high quality color consistency results on large photo collections of popular tourist landmarks and personal photo collections containing images of people.