The stress response in Drosophila melanogaster reveals sex differences in behavior, similar to what has been observed in mammals. However, unlike mammals, the sex determination pathway in Drosophila is well established, making this an ideal system to identify factors involved in the modulation of sex-specific responses to stress. In this study, we show that the Drosophila fat body, which has been shown to be important for energy homeostasis and sex determination, is a dynamic tissue that is altered in response to stress in a sex and time dependent manner. We manipulated the sex determination pathway in the fat body via targeted expression of transformer and transformer-2, and analyzed these animals for changes in their response to stress. In the majority of cases, manipulation of transformer or transformer-2 was able to change the physiological output in response to starvation and oxidative stress to that of the opposite sex. Our data also uncover the possibility of additional downstream targets for transformer and transformer-2 that are separate from the sex determination pathway and can influence behavioral and physiological responses.