Face recognition abilities improve between adolescence and adulthood over typical development (TD), but plateau in autism, leading to increasing face recognition deficits in autism later in life. Developmental differences between autism and TD may reflect changes between neural systems involved in the development of face encoding and recognition. Here, we focused on whole‐brain connectivity with the fusiform face area (FFA), a well‐established face‐preferential brain region. Older children, adolescents, and adults with and without autism completed the Cambridge Face Memory Test, and a matched car memory test, during fMRI scanning. We then examined task‐based functional connectivity between the FFA and the rest of the brain, comparing autism and TD groups during encoding and recognition of face and car stimuli. The autism group exhibited underconnectivity, relative to the TD group, between the FFA and frontal and primary visual cortices, independent of age. Underconnectivity with the medial and rostral lateral prefrontal cortex was face‐specific during encoding and recognition, respectively. Conversely, underconnectivity with the L orbitofrontal cortex was evident for both face and car encoding. Atypical age‐related changes in connectivity emerged between the FFA and the R temporoparietal junction, and R dorsal striatum for face stimuli only. Similar differences in age‐related changes in autism emerged for FFA connectivity with the amygdala across both face and car recognition. Thus, underconnectivity and atypical development of functional connectivity may lead to a less optimal face‐processing network in the context of increasing general and social cognitive deficits in autism. Underconnectivity in addition to atypical age‐related changes in functional connectivity between key face‐processing regions and domain‐general regions may underlie atypical development of face recognition abilities in autism.