Male hypogonadism results in changes in body composition characterized by increases in fat mass. Resident immune cells influence energy metabolism in adipose tissue and could promote increased adiposity through paracrine effects. We hypothesized that manipulation of circulating sex steroid levels in healthy men would alter adipose tissue immune cell populations. Subjects (n=44 men, 19-55 years of age) received 4 weeks of treatment with the GnRH receptor antagonist acyline with daily administration of 1) placebo gel, 2) 1.25g testosterone gel (1.62%), 3) 5g testosterone gel, or 4) 5g testosterone gel with an aromatase inhibitor. Subcutaneous adipose tissue biopsies were performed at baseline and end-of-treatment, and adipose tissue immune cells, gene expression, and intra-adipose estrogen levels were quantified. Change in serum total testosterone level correlated inversely with change in the number of CD3+ (β=-0.36, p=0.04), CD4+ (β=-0.34, p=0.04), and CD8+ (β=-0.33, p=0.05) T cells within adipose tissue. Change in serum 17β-estradiol level correlated inversely with change in the number of adipose tissue macrophages (ATMs) (β=-0.34, p=0.05). A negative association also was found between change in serum testosterone and change in CD11c+ ATMs (β=-0.41, p=0.01). Overall, sex steroid deprivation was associated with increases in adipose tissue T cells and ATMs. No associations were found between changes in serum sex steroid levels and changes in adipose tissue gene expression. Circulating sex steroid levels may regulate adipose tissue immune cell populations. These exploratory findings highlight a possible novel mechanism that could contribute to increased metabolic risk in hypogonadal men.