Who, in particular, may hold us responsible for our moral failings? Most discussions of moral responsibility bracket this question, despite its obvious practical importance. In this article, I investigate the moral authority involved and how it arises in the context of personal relationships, such as friendship or family relations. My account is based on the idea that parties to a personal relationship not only share responsibility for their relationship, but also — to some degree that is negotiated between them — for one another's lives. In sharing these responsibilities, we grant people a particular authority to respond to us. By highlighting the responsibility that we assume when we hold someone responsible, I also suggest that this analysis contains important lessons for thinking about responsibility in other contexts.