Single-shot structured light scanning has been actively investigated as it can recover accurate geometric shapes even in dynamic scenes. Since many single-shot approaches focus on improving depth accuracy, recovering the intrinsic properties of the scene, such as albedo and shading, is also valuable. In this paper, we propose a novel method that reconstructs not only the metric depth but also the intrinsic properties from a single structured light image. We extend the conventional color structured light model to embrace the Lambertian shading model. By using a color phaseshifting pattern, we parameterize the captured image with only two variables, albedo and depth. For an initial solution, a simple but powerful method to decompose sinusoids from the input image is presented. We formulate a nonlinear cost function and jointly optimize albedo and depth efficiently by calculating the analytic Jacobian. We demonstrate that our algorithm works reasonably on various realworld objects that exhibit challenging surface reflectance and albedo.