Debates about slavery, segregation, and racial inequality in the United States were often bound up with the meanings of racial “prejudice.” In this article, I suggest that the concept was often double-edged: deployed both against racial inequality and oppression, but also to maintain it. Since the end of the eighteenth century, abolitionists and other advocates of racial equality charged that their opponents were possessed by irrational prejudice, which they sought to stamp out through a variety of means. In another line of argument, however, racial prejudice was natural or at least so deeply rooted from centuries of slavery as to be basically ineradicable. This meant that attempts to abolish slavery and establish an egalitarian, multiracial society were forever doomed to failure. Some people drew the lesson from this conception of prejudice that it might be best to remove blacks from American soil altogether by colonizing them elsewhere, particularly West Africa. Abolitionists, however, did not accept the idea that racial prejudice was invincible and thought it could be removed through greater education. After the Civil War, with the end of slavery, defenders of segregation drew on similar arguments, suggesting that if there were prejudices between the races, these resulted from the wisdom of the ages and should be respected, even as supporters of racial equality sought to show that these prejudices need not be permanent. This article therefore explores the complex and sometimes counterintuitive uses of the concept of racial “prejudice” from the late eighteenth century up until the subsequent development of the Jim Crow segregation regime in the late nineteenth century.