Although race-based discrimination and stereotyping can only occur if people place others into racial categories, our understanding of this process, particularly in contexts where observers categorize others based solely on appearance, is limited. Using a unique data set drawn from observers’ assessments of photos posted by White, Black, Latino, and multiracial online daters, this study examines how phenotype and observer characteristics influence racial categorization and cases of divergence between self-identities and others’ classifications. I find that despite the growth in the multiracial population, observers tend to place individuals into monoracial categories, including Latino. Skin color is the primary marker used to categorize others by race, with light skin associated with Whiteness, medium skin with Latinidad, and, most strongly, dark skin with Blackness. Among daters who self-identify as Black along with other racial categories, those with dark skin are overwhelmingly placed solely into a Black category. These findings hold across observers, but the proportion of photos placed into different racial categories differs by observers’ gender and race. Thus, estimates of inequality may vary depending not only on how race is assessed but also on who classifiers are. I argue that patterns of racial categorization reveal how the U.S. racial structure has moved beyond binary divisions into a system in which Latinos are seen as a racial group in-between Blacks and Whites, and a dark-skin rule defines Blacks’ racial options.