Objective Pay‐for‐performance (P4P) is commonly used to improve health care quality in the United States and is expected to be frequently implemented under the Affordable Care Act. However, evidence supporting its use is mixed with few large‐scale, rigorous evaluations of P4P. This study tests the effect of P4P on quality of care in a large‐scale setting—the implementation of P4P for nursing homes by state Medicaid agencies. Data Sources/Study Setting 2001–2009 nursing home Minimum Data Set and Online Survey, Certification, and Reporting (OSCAR) datasets. Study Design Between 2001 and 2009, eight state Medicaid agencies adopted P4P programs in nursing homes. We use a difference‐in‐differences approach to test for changes in nursing home quality under P4P, taking advantage of the variation in timing of implementation across these eight states and using nursing homes in the 42 non‐P4P states plus Washington, DC as contemporaneous controls. Principal Findings Quality improvement under P4P was inconsistent. While three clinical quality measures (the percent of residents being physically restrained, in moderate to severe pain, and developed pressure sores) improved with the implementation of P4P in states with P4P compared with states without P4P, other targeted quality measures either did not change or worsened. Of the two structural measures of quality that were tied to payment (total number of deficiencies and nurse staffing) deficiency rates worsened slightly under P4P while staffing levels did not change. Conclusions Medicaid‐based P4P in nursing homes did not result in consistent improvements in nursing home quality. Expectations for improvement in nursing home care under P4P should be tempered.