The self-assembly of alternative conformations of normal proteins into amyloid aggregates has been implicated in both the acquisition of new functions and in the appearance and progression of disease. However, while these amyloidogenic pathways are linked to the emergence of new phenotypes, numerous studies have uncoupled the accumulation of aggregates from their biological consequences, revealing currently underappreciated complexity in the determination of these traits. Here, to explore the molecular basis of protein-only phenotypes, we focused on the Saccharomyces cerevisiae Sup35/[PSI(+)] prion, which confers a translation termination defect and expression level-dependent toxicity in its amyloid form. Our studies reveal that aggregated Sup35 retains its normal function as a translation release factor. However, fluctuations in the composition and size of these complexes specifically alter the level of this aggregate-associated activity and thereby the severity and toxicity of the amyloid state. Thus, amyloid heterogeneity is a crucial contributor to protein-only phenotypes.