In nanoindentation, roughness of the sample surface can be a severe source of error in the determination of properties from indentation tests. Recently, roughness was also considered as a crucial issue in understanding the indentation size effect where a significant increase in hardness was seen with the decrease of depth. A three-dimensional roughness model with the Johnson–Cook material model is employed to study the roughness effect in nanoindentation on AISI316L stainless steel by use of finite element method. The rough surface is obtained by generating a random function in Matlab and then applying fast Fourier transform. With the quantitative analysis the mechanical properties such as the hardening and variation of the reduced modulus are found. From both the experimental and simulation results, the hardness distribution shows strengthening effect with the increased surface roughness. Both the scatter of hardness and indentation modulus increases with the increased roughness. In addition, the dependence of the pile-up effect and the contact area on the roughness is studied and analyzed.