Objective There is increasing interest in identifying high‐quality physicians, such as whether physicians perform above or below a threshold level. To evaluate whether current methods accurately distinguish above‐ versus below‐threshold physicians, we estimate misclassification rates for two‐category identification systems. Data Sources Claims data for Medicare fee‐for‐service beneficiaries residing in Florida or New York in 2010. Study Design Estimate colorectal cancer, glaucoma, and diabetes quality scores for 23,085 physicians. Use a beta‐binomial model to estimate physician score reliabilities. Compute the proportion of physicians whose performance tier would be misclassified under three scoring systems. Principal Findings In the three scoring systems, misclassification ranges were 8.6–25.7 percent, 6.4–22.8 percent, and 4.5–21.7%. True positive rate ranges were 72.9–97.0 percent, 83.4–100.0 percent, and 34.7–88.2 percent. True negative rate ranges were 68.5–91.6 percent, 10.5–92.4 percent, and 81.1–99.9 percent. Positive predictive value ranges were 70.5–91.6 percent, 77.0–97.3 percent, and 55.2–99.1 percent. Conclusions Current methods for profiling physicians on quality may produce misleading results, as the number of eligible events is typically small. Misclassification is a policy‐relevant measure of the potential impact of tiering on providers, payers, and patients. Quantifying misclassification rates should inform the construction of high‐performance networks and quality improvement initiatives.