To achieve a synergistic effect of various fibers in hot-mix asphalt, an optimization method for determining the optimal composition ratio of hybrid fiber (HF), containing lignin, polyester, and basalt fibers (BFs) was presented according to the simplex centroid design method and the rheological properties of HF-reinforced asphalt mortar. Additionally, the pavement performance and dynamic rheological characteristics of the HF-reinforced stone matrix asphalt (SMA) were assessed using the wheel tracking test, beam bending test, freeze–thaw indirect tensile test, and dynamic modulus test (DMT). The results indicate that the optimal composition ratio of lignin, polyester, and BFs in HF is 0:0.55:0.45, respectively. Furthermore, HF-reinforced SMA demonstrates the highest resistance to moisture damage and rutting among the fiber-reinforced SMAs, with low-temperature cracking resistance comparable to that of polyester fiber-reinforced SMA. The DMT indicate that the HF-reinforced SMA has the largest dynamic modulus among the four SMAs, suggesting that it exhibits the strongest stiffness and resistance to deformation. Additionally, master curves of the dynamic modulus were constructed for the various fiber-reinforced SMAs.