In order to study the residual shear fatigue strength of asphalt mixture，a two-dimensional numerical model of shear test was established based on the micro-structure of asphalt mixture and the multi-scale algorithm. According to the multi-scale algorithm and the finite element numerical simulation，the fatigue residual strength of asphalt mixture was predicted gradually，and that was compared with the shear residual strength of the test. The results show that the meso-mechanical scale model of asphalt mixture can reflect the influence of matrix and aggregate on the fatigue damage of asphalt mixture. The residual shear fatigue strength of asphalt mixture is predicted gradually by the multi-scale algorithm，which can avoid the absence of aggregate in the modeling process. The error between the fatigue shear strength obtained by the asphalt mixture shear test and the predicted value of the multi-layer iterative algorithm is relatively stable，which meets the engineering requirements.