Recently, various material recognition methods have been introduced that use a single color or light field camera. In prior methods, color and texture information of an object are used as key features. However, there exists fundamental limitation in using color features for material recognition in that material type can be characterized better by surface reflectance, visual appearance rather than its color and textures. In this work, we propose IR surface reflectance based material type recognition method. We use off-the-shelf ToF camera to estimate the IR reflectance of arbitrary surface. Material type recognition is performed on both color and surface IR reflectance features. Several network structures including gradual convolutional neural network are proposed and verified for our material recognition within our own 3D data sets.