The progression of an infection within a host determines the ability of a pathogen to transmit to new hosts and to maintain itself in the population. While the general connection between the infection dynamics within a host and the population-level transmission dynamics of pathogens is widely acknowledged, a comprehensive and quantitative understanding that would allow full integration of the two scales is still lacking. Here, we provide a brief discussion of both models and data that have attempted to provide quantitative mappings from within-host infection dynamics to transmission fitness. We present a conceptual framework and provide examples of studies that have taken first steps towards development of a quantitative framework that scales from within-host infections to population-level fitness of different pathogens. We hope to illustrate some general themes, summarize some of the recent advances and-maybe most importantly-discuss gaps in our ability to bridge these scales, and to stimulate future research on this important topic.