Rapid progress in information and communication technologies (ICTs) has fueled the popularity of e‐learning. However, an e‐learning environment is limited in that online instructors cannot monitor immediately whether students remain focus during online autonomous learning. Therefore, this study tries to develop a novel attention aware system (AAS) capable of recognizing students' attention levels accurately based on electroencephalography (EEG) signals, thus having high potential to be applied in providing timely alert for conveying low‐attention level feedback to online instructors in an e‐learning environment. To construct AAS, attention responses of students and their corresponding EEG signals are gathered based on a continuous performance test (CPT), ie, an attention assessment test. Next, the AAS is constructed by using training and testing data by the NeuroSky brainwave detector and the support vector machine (SVM), a well‐known machine learning model. Additionally, based on the discrete wavelet transform (DWT), the collected EEG signals are decomposed into five primary bands (ie, alpha, beta, gamma, theta, and delta). Each primary band contains five statistical parameters (including approximate entropy, total variation, energy, skewness, and standard deviation), thus generating 25 potential brainwave features associated with students' attention level for constructing the AAS. An attempt based on genetic algorithm (GA) is also made to enhance the prediction performance of the proposed AAS in terms of identifying students' attention levels. According to GA, the seven most influential features are selected from 25 considered features; parameters of the proposed AAS are also optimized. Analytical results indicate that the proposed AAS can accurately recognize individual student's attention state as either a high or low level, and the average accuracy rate reaches as high as 89.52%. Moreover, the proposed AAS is integrated with a video lecture tagging system to examine whether the proposed AAS can accurately detect students' low‐attention periods while learning about electrical safety in the workplace via a video lecture. Four experiments are designed to assess the prediction performance of the proposed AAS in terms of identifying the periods of video lecture with high‐ or low‐attention levels during learning processes. Analytical results indicate that the proposed AAS can accurately identify the low‐attention periods of video lecture generated by students when engaging in a learning activity with video lecture. Meanwhile, the proposed AAS can also accurately identify the low‐attention periods of video lecture generated by students to some degree even when students engage in a learning activity by a video lecture with random disturbances. Furthermore, strong negative correlations are found between the students' learning performance (ie, posttest score and progressive score) and the low‐attention periods of video lecture identified by the proposed AAS. Results of this study demonstrate that the proposed AAS is effective, capable of assisting online instructors in evaluating students' attention levels to enhance their online learning performance.