Purpose Rural‐dwelling persons with HIV infection often have limited access to HIV specialty care, and they may instead use more nearby primary care. This study described use of infectious disease (ID) specialty and general primary care services among rural compared with urban veterans with HIV in the United States and determined associations between geographic access to ID and primary care and use of care. Methods The sample included all veterans in the national Veterans Administration (VA) HIV clinical case registry in 2009 (N = 23,669, 10.2% rural). Geographic access was measured by calculating travel times to the nearest VA primary care and ID specialty clinic. Findings Rural veterans were less likely than urban to use ID clinics (82% of rural vs 87% of urban, P < .01) and more likely to use primary care (82% vs 73%, P < .01). As travel time to ID care increased from less than 15 minutes to over 90 minutes, use of ID care decreased from 88% to 71% (P < .01), while use of primary care increased from 68% to 86% (P < .0001). In multivariable models, increased travel time to ID care—but not rural residence—was associated with decreased ID and increased primary care use. Conclusions Persons with HIV who live far from ID specialty clinics are less likely to use specialty care and more likely to use primary care. Specialty clinics should consider using telehealth to deliver care over distance and programs to coordinate “shared care” relationships with distant primary care providers.