Driver drowsiness greatly increases the driver’s risk of a crash or near-crash. It is recognized as one of the major causes of severe traffic accidents. In this paper, a novel non-intrusive surveillance system is proposed to estimate driver drowsiness by fusion of visual information about lane and driver with Dempster–Shafer theory. Based on expert knowledge and data statistics, various visual features extracted from lane and eye tracking are analysed for their correlation with driver drowsiness in the framework of the subjective ‘observer rating of drowsiness’. The system is validated in real road scenarios and the experiment results demonstrate that it is promising in improving the robustness and temporal response of driver surveillance in real time.