Driver inattention, either driver drowsiness or distraction, is a major contributor to serious traffic crashes. In general, most research on this topic studies driver drowsiness and distraction separately, and is often conducted in a well-controlled, simulated environment. By considering the reliability and flexibility of real-time driver monitoring systems, it is possible to evaluate driver inattention by the fusion of multiple selected cues in real life scenarios. This paper presents a real-time, visual-cue-based driver monitoring system, which can track both multi-level driver drowsiness and distraction simultaneously. A set of visual cues are adopted via analysis of drivers’ physical behaviour and driving performance. Driver drowsiness is evaluated using a multi-level scale, by applying evidence theory. Additionally, a general framework of extensive hierarchical combinations is used to generate a probabilistic evaluation of driving risk in real time. This driver inattention monitoring system with multimodal fusion has been proven to improve the accuracy of risk evaluation and reduce the rate of false alarms, and acceptance of the system is recommended.