This study aimed to propose a load evaluation index based on eye movement data—eye movement dispersion—and to determine the thresholds of eye movement dispersion under different visual loads. The index was then used to explore the ranking of the influence of various traffic environment factors on driver visual load, providing reference for traffic design. The research utilized the DADA2000 dataset and employed an eye movement feature recognition algorithm to obtain the degree of eye movement dispersion. By distinguishing the scenes before and after accidents, the thresholds of eye movement dispersion under different visual loads were determined. Furthermore, a semantic segmentation algorithm was used to match the concentrated areas of eye movement points within 3 seconds before the accident with the elements obtained through semantic segmentation. A machine learning algorithm was then used to rank the importance of each traffic environment element. The study found that using eye movement dispersion to distinguish driver visual load was feasible. The eye movement dispersion decreased to one-third of its original value shortly before an accident, and the eye movement points concentrated on collision objects or the main responsible factors of the accident. The primary elements contributing to driver visual load were bicycle participants, roadside marker posts, and road conditions. These findings could be further utilized for accident prevention and prediction. The study concluded that visual load was most affected by traffic elements that invade the right of way (e.g., non-motor vehicles occupying motor vehicle lanes) or unclear right of way (e.g., roads without markings or with fuzzy markings). In practical traffic design, the allocation of right of way for each participant should be prioritized, ensuring that the right of way is strictly maintained without interference.