It is well known that human beings do not always change opinions or attitudes, since the duration of interaction with others usually has a significant impact on one's decision-making. Based on this observation, we introduce a freezing period into the voter model, in which the frozen individuals have a weakened opinion switching ability. We unfold the presence of an optimal freezing period, which leads to the fastest consensus, using computation simulations as well as theoretical analysis. We demonstrate that the essence of an accelerated consensus is attributed to the biased random walk of the interface between adjacent opinion clusters. The emergence of an optimal freezing period is robust against the size of the system and the number of distinct opinions. This study is instructive for understanding human collective behavior in other relevant fields.