Many bacterial and archaeal lineages have a history of extensive and ongoing horizontal gene transfer and loss, as evidenced by the large differences in genome content even among otherwise closely related isolates. How ecologically cohesive populations might evolve and be maintained under such conditions of rapid gene turnover has remained controversial. Here we synthesize recent literature demonstrating the importance of habitat and niche in structuring horizontal gene transfer. This leads to a model of ecological speciation via gradual genetic isolation triggered by differential habitat-association of nascent populations. Further, we hypothesize that subpopulations can evolve through local gene-exchange networks by tapping into a gene pool that is adaptive towards local, continuously changing organismic interactions and is, to a large degree, responsible for the observed rapid gene turnover. Overall, these insights help to explain how bacteria and archaea form populations that display both ecological cohesion and high genomic diversity.