The aims of this study were to examine associations between reported exposure to psychological and physical violence and psychological distress (PD) among men and women, and to explore the possible mediating or moderating roles of sense of mastery and social support. We used data from the nationally representative Norwegian Health and Level of Living Survey in 2005 and 2012 (Weighted N = 19,386). PD was measured with the Hopkins Symptom Checklist–25, using the subscales for anxiety and depression separately and in combination. Analyses were conducted using hierarchical logistic regression with complex sample adjustment. Altogether, 3.8% of men and 5.4% of women reported psychological violence during the last 12 months, while 2.3% and 1.6% reported physical violence, respectively. Both forms of violence were associated with excess risk of comorbid anxiety and depressive symptoms above clinical cut-point (CAD) in men and women alike, and CAD occurred more frequently than anxiety or depressive problems separately. Sense of mastery, but not social support, partly mediated the association between both forms of violence and CAD in men, whereas both partly mediated the association between psychological violence and CAD in women. No moderator role was indicated. Overall, the results provide evidence for excess risk of PD, particularly CAD, in men and women reporting exposure to violence. Sense of mastery and to a lesser degree social support were shown to constitute significant mediators, underscoring the importance of systems for strengthening coping strategies and social support among violence victims, such as psychological and practical support by the health services.