Objective Trauma and depression are associated with brain structural alterations; their combined effects on these outcomes are unclear. We previously reported a negative effect of trauma, independent of depression, on verbal learning and memory; less is known about underlying structural associates. We investigated separate and interactive associations of trauma and depression on brain structure. Methods Adults aged 30–89 (N = 203) evaluated for depression (D+) and trauma history (T+) using structured clinical interviews were divided into 53 D+T+, 42 D+T−, 50 D−T+, and 58 D−T−. Multivariable linear regressions examined the separate and interactive associations of depression and trauma with prefrontal and temporal lobe cortical thickness composites and hippocampal volumes adjusting for age, sex, predicted verbal IQ, comorbid anxiety, and vascular risk. Significant results informed analyses of tract‐based structural connectomic measures of efficiency and centrality. Results Trauma, independent of depression, was associated with greater left prefrontal cortex (PFC) thickness, in particular the medial orbitofrontal cortex and pars orbitalis. A trauma × depression interaction was observed for the right PFC in age‐stratified analyses: Older D + T+ had reduced PFC thickness compared with older D − T+ individuals. Regardless of age, trauma was associated with more left medial orbitofrontal cortex efficiency and less pars orbitalis centrality. In the T+ group, left pars orbitalis cortical thickness and centrality negatively correlated with verbal learning. Conclusions Trauma, independent of depression, associated with altered PFC characteristics, morphologically and in terms of structural network communication and influence. Additionally, findings suggest that there may be a combined effect of trauma and depression in older adults. Copyright © 2017 John Wiley & Sons, Ltd.