The evolutionary rates of RNA viruses can differ from one another by several orders of magnitude. Much of this variation has been explained by differences in viral mutation rates and selective environments. However, substitution rates also vary considerably across viral populations belonging to the same species. In particular, viral lineages from epidemic regions tend to have higher substitution rates than those from endemic regions, and lineages from populations with higher contact rates tend to have higher substitution rates than those from populations with lower contact rates. We address the mechanism behind these patterns by using a nested modeling approach, whereby we integrate within-host viral replication dynamics with a population-level epidemiological model. Through numerical simulations and analytical approximations, we show that variation in viral substitution rates over the course of an infection, coupled with differences in age of infection of transmitting hosts under different epidemiological scenarios, can explain these evolutionary patterns. We further derive analytical estimates of expected substitution rate differences under epidemic versus endemic epidemiological conditions. By comparing these estimates to empirical data for four viral species, we show that these factors are sufficient to explain observed variation in substitution rates in three of four cases. This work shows that even in neutrally evolving viral populations, epidemiological dynamics can alter substitution rates via the interplay between within-host replication dynamics and population-level disease dynamics.