Ultrasonic fatigue tests were carried out on the aluminum alloy AISI 6061-T6, presenting different values for the principal surface roughness parameters: R_a, R_q, and R_z. For fatigue life comprised between 3 × 10⁵ ∼ 6 × 10⁶ cycles, crack initiates at the specimen surface induced by stress-concentration and micro-plastic deformation (micro-void coalescence); whereas for the very high cycle fatigue (>10⁷ cycles), the mechanism of crack initiation moves to subsurface or inside the specimen and is associated with internal imperfections such as micro-porosities or nonmetallic inclusions. For the first fatigue life regime, compression residual stresses induced by the work-hardening machining process and the associated micro-plastic deformation are the principal factors controlling the fatigue endurance on this aluminum alloy. Experimental results show that fatigue endurance is higher for the high surface roughness in the 3 × 10⁵ ∼ 6 × 10⁶ cycles of fatigue life, whereas this behavior is inversed in the very high-cycle fatigue regime. These results are analyzed in terms of residual compression stresses induced by the work-hardening machining process, the surface roughness, and the reverse yielding or Bauschinger effect.