Infection by distinct Dengue virus serotypes and host immunity are intricately linked. In particular, certain levels of cross-reactive antibodies in the host may actually enhance infection severity leading to Dengue hemorrhagic fever (DHF). The coupled immunological and epidemiological dynamics of Dengue calls for a multi-scale modeling approach. In this work, we formulate a within-host model which mechanistically recapitulates characteristics of antibody dependent enhancement in Dengue infection. The within-host scale is then linked to epidemiological spread by a vector-host partial differential equation model structured by host antibody level. The coupling allows for dynamic population-wide antibody levels to be tracked through primary and secondary infections by distinct Dengue strains, along with waning of cross-protective immunity after primary infection. Analysis of both the within-host and between-host systems are conducted. Stability results in the epidemic model are formulated via basic and invasion reproduction numbers as a function of immunological variables. Additionally, we develop numerical methods in order to simulate the multi-scale model and assess the influence of parameters on disease spread and DHF prevalence in the population.