The towers of special-shaped cable-stayed bridges have low stiffness and complex stress conditions, with strong aerodynamic coupling phenomena between the girder and the towers. Current research on the impact of wind effects on bridge buffeting responses primarily focused on conventional multi-tower cable-stayed bridges and suspension bridges, with few studies addressing the wind effects on towers of special-shaped cable-stayed bridges. In this paper, wind tunnel experiments and numerical simulations were used to obtain three-component coefficients of the girder and tower sections of a special-shaped cable-stayed bridge, as well as the fluctuating wind speed time histories. Based on the quasi-steady theory, a time-domain model of the aerodynamic self-excited forces was developed. On this basis, the influences of self-excited forces and wind effects on the buffeting response of the bridge were studied separately. The results indicate that the vertical displacement of the girder is minimally affected by the wind effects on the towers, while the lateral displacement response of the towers and the lateral and torsional displacement responses of the girder are significantly influenced by the wind effects, with a noticeable increase in the lateral displacement response. Additionally, the self-excited forces slightly reduce the displacement response of the girder, but the impact is minimal.