The rolling bottle test (RBT) was used to investigate the adhesion of four types of aggregates (slate, limestone, granite, and diabase) to both base asphalt and polymer-modified asphalt (PMA). Additionally, the RBT was applied to evaluate the improvement effects of various antistripping measures on the adhesion performance of slate aggregate. The test results indicated that the RBT was capable of quantitatively assessing the adhesion between various types of aggregates and asphalt binder, the feasibility of applying slate aggregate in asphalt concrete, and the effectiveness of antistripping agents. The high quartz content in slate can result in poor adhesion between slate aggregate and asphalt binder. PMA was found to be the optimal option for improving the adhesion between slate aggregate and asphalt. At the same time, hydrated lime was the most effective antistripping agent when base asphalt was used as the binder. Cement and liquid antistripping agent only had limited improvement effects. The poor adhesion between slate aggregate and base asphalt could not be fundamentally solved by means of antistripping agents. Further tests are needed on an asphalt mixture containing slate aggregate to evaluate its susceptibility to water damage.