Background  People with intellectual disabilities (ID) receive primary care in community‐based practices and are encouraged to participate in the physician–patient relationship. However, the nature of this participation is not known. Methods  Qualitative data were analysed to obtain perspectives from patients and providers regarding clinic visits. Patient participants were recruited from community organisations, while physician participants were recruited from emails and phone calls to local and regional practices and a national and regional list serve. Analysis methods derived from grounded theory were used. Results  Twenty‐seven women with ID and 22 family physicians were interviewed. Themes important to both groups included time, how the support worker should be used in the encounter and the nature of the physician–patient relationship. Patients expressed frustration at how little time they spent with their physician, and wished that physicians would speak directly to them instead of to their support worker. Physicians felt that patients with ID took too much time, and said that they preferred communicating with the support worker. The interviews also revealed unconscious biases about people with ID. Conclusions  Patient participation is encouraged for people with ID, but is limited because of both physician and patient factors. Greater awareness of these factors may improve care for patients with ID.