As one of the constitutive features of the interface, the surface geometry has a significant influence on the shear behavior. To investigate the influence of surface roughness on a granular material/structure interface, both the direct shear test under constant normal stress and simple numerical modeling were carried out. Sand with colored pillars were used to provide a three-dimensional observation method to record the thickness variation and spatial distribution of shear zone. The results indicate that the spatial distribution of the shear zone is not uniform, showing a tendency toward a thick center and thin ends in the shear direction; the thickness of the shear zone is between 8–14 times the average particle size. With the support of numerical modeling, it can be concluded that the development mechanism of the shear zone changes as the interface roughness increases. The mosaic effect dominates the shear deformation when the normalized roughness parameter (R_n) is 1.25, whereas it is the compact effect for much larger R_n.