Research on the effectiveness of competence-based education (CB-education) across educational contexts and levels requires a new evaluation measurement. This study explores the face validity, construct validity, and robustness of a competency self-report instrument that is aligned with contemporary competence theory and with current educational practice based on CB-qualification frameworks. A pilot study showed face validity of the competency constructs and indicators according to students from various levels in tertiary education. The results of the principal components analyses and parallel analyses, using data from 351 secondary vocational education and academic students, show more construct validity and robustness for competency constructs that are concrete and easy to relate to specific situations (e.g., "applying expertise") compared with the abstract competencies (e.g., "deciding and initiating"). This article sets out implications for designing and administrating uniform competency self-reports across educational levels and discusses suggestions for subsequent research.