Understanding the frictional behavior of coarse-grained soils was important for improving compaction in geotechnical engineering. Nonetheless, the influence of particle shape and surface roughness on friction and compaction across varying loading conditions remains inadequately understood, thereby constraining the development of effective soil compaction methods. This study investigated the friction coefficient (μ) of two types of coarse-grained soils, pebble and rubble, by considering particle roughness (R_g), sliding velocity (V_r), normal pressure (F_N), and dry density (ρ^d) through vibration compaction tests. Moreover, Discrete Element Method (DEM) was employed to verify the effects of F_N, V_r, and ρ^d on particle R_g under various loads. The results showed that pebbles with smoother surfaces achieved higher dry densities and more efficient particle rearrangement, whereas rubbles with rougher and more angular particles compacted more slowly. Lower sliding velocities increased friction coefficients, whereas higher velocities reduced friction, with surface roughness playing a key role. Increased normal pressure lowered frictional resistance and stabilized μ, enhancing mechanical stability. Specifically, μ decreased by approximately 20 % as the sliding velocity increased from 1 to 20 mm/s and by approximately 15 % as the normal force (F_n) increased from 1 to 5 N. Moreover, the DEM simulations closely matched laboratory results, including lower friction coefficients (μ = 0.1–0.3), which enabled faster particle rearrangement and more efficient compaction. In contrast, higher coefficients (μ = 0.5–0.9) slowed compaction because of increased resistance. These findings demonstrate the practical value of combining experimental and numerical approaches to predict and optimize soil compaction behavior more effectively.