Despite the proliferation of academic studies examining financial literacy and financial outcomes, no consistent definition or empirically validated measures of financial literacy exist. While a handful of questions have become the standard measures of financial literacy in previous research, little work has been done examining whether responses to these questions accurately capture underlying financial capability, or whether they causally relate to subsequent financial well‐being. Taking advantage of longitudinal data from the Health and Retirement Study we examine whether some of the questions previously used as measures of financial literacy are consistent measures of financial knowledge and effective predictors of future changes in wealth. We find that respondents frequently do not consistently answer questions across survey waves and that the context in which a question is asked affects the likelihood of correctly responding. Moreover, our regression analyses suggest that correctly answering these questions, consistently or not, has little significant relationship to changes in wealth over time, and is often related to a decrease in future wealth. Our findings should give pause to researchers using the financial literacy questions examined here, particularly from cross‐sectional data.