Physical distancing has been the primary strategy to control COVID-19 in the United States. We used mobility data from a large, anonymized sample of smartphone users to assess the relationship between neighbourhood income and physical distancing during the pandemic. We found a strong gradient between neighbourhood income and physical distancing. Individuals in high-income neighbourhoods increased their days at home substantially more than individuals in low-income neighbourhoods did. Residents of low-income neighbourhoods were more likely to work outside the home, compared to residents in higher-income neighbourhoods, but were not more likely to visit locations such as supermarkets, parks and hospitals. Finally, we found that state orders were only associated with small increases in staying home in low-income neighbourhoods. Our findings indicate that people in lower-income neighbourhoods have faced barriers to physical distancing, particularly needing to work outside the home, and that state physical distancing policies have not mitigated these disparities.