This is a question that I get asked all the time, so I thought I would spend a few hours delving into some recent academic research to see what I could find. Answering this question in an analytically rigorous way becomes more imperative given the multiplicity of financial literacy curriculums and approaches being developed by school districts and financial firms alike. How do we know which ones work?
I would summarize the answer provided by these Fed researchers who evaluated the existing academic literature as "we don't know now, but we do have some suggestions on how to figure it out:"
"Taken together, the literature does not succeed in establishing the extent of the benefit provided by financial education programs, nor does it provide conclusive support that any benefit at all exists. The reason is not clear, but could only be one of three culprits: either (a) financial education simply does not work, (b) financial education does work but the programs are not effective at transferring knowledge (i.e. poorly designed programs), or (c) financial education does work and the programs are properly designed, but program evaluation techniques are not yet adequate in capturing these effects. We believe that option (a) is unlikely, and that some combination of (b) and (c) is more likely. Either way, it is clear that more extensive program evaluation and further research are much needed."
The researchers make the following two suggestions about how to increase the impact of financial literacy or financial education programs (hint: provide education just prior to decision being made):
"First, while the overall evidence in favor of financial education remains unclear, we do see a
pattern that highly targeted programs, unlike general programs, tend to be effective in changing people’s financial behavior, both in the short run and the long run. As a result, we contend that programs should be highly targeted toward a specific audience and area of financial activity (e.g. home-ownership or credit card counseling, etc.), and that this training occurs just before the corresponding financial event (e.g. purchase of a home or use of a credit card, etc.). Secondly, we recommend that including formal program evaluation methods in the design of the program itself is critical in being able to measure whether the programs are achieving intended outcomes."
So, what would an effective financial literacy evaluation look like (bold is mine and I reformatted to include bullet points)?
"...we believe it is possible to define a standard but adaptable framework that will accommodate all types of literacy programs. Fox, Bartholomae and Lee (2005) provide one such framework, which has been tested in the MoneyMinded program in Australia.
The authors describe five major steps they believe should be included when evaluating financial education programs: pre-implementation (and needs assessment), accountability, program clarification, progress towards objectives, program impact.
- In the pre-implementation stage, the target group is identified, needs are assessed, and goals are specified. Administering literacy tests on the target group is a good proxy for a needs assessment. General indicators of needs could be high rates of non-business bankruptcy filings, defaults on loans, and high consumer debt levels, among others.
- The accountability stage involves the collection of information on education and services provided, program cost, and basic information on program participants. The objective here is to determine who has been reached by the program and in what way; that is, whether the population in need is the population actually served.
- The program clarification stage helps the program planners review an ongoing program’s goals and objectives and assess whether these goals and objectives should be revised.
- Next, the progress towards objectives stage involves obtaining objective measures (quantified data) of the impact of the program on the participants, and how those impacts relate to program goals.
- Finally, the program impact stage involves an experimental approach (comparing sample and control groups) to assess both the short-term and long-term effects of the program. Information collected in the previous stage (progress towards objectives) helps assess whether there were long-term and short-term effects. According to the authors, there is scarce evidence of evaluation of financial literacy programs at the final stage (program impact) because most financial education programs do not include impact evaluation as a component of their program design."
Lauren E. Wills in Evidence and Ideology in Assessing the Effectiveness of Financial Literacy Education cites widely held beliefs in the value of financial literacy education (FLE), but little analytical support demonstrating its effectiveness (bold is mine):
"FLE is widely believed to turn consumers into “responsible” and “empowered” market players, motivated and competent to engage in financial behaviors that increase their own welfare (Greenspan 2002). Ideally, educated consumers handle their own credit, insurance, savings and investment matters by confidently navigating the marketplace. In this idealized world, substantive legal regulation of financial products is unnecessary and even counterproductive. This vision depends on the belief that FLE can not only improve financial behavior, but that it can do so to the degree necessary for consumers to protect and even increase their welfare in the modern financial marketplace...Although cited by policymakers as support for financial literacy initiatives (e.g., U.S. Senate Bill 2005, citing Danes 2004), research to date has yet to produce reliable,statistically significant evidence of effectiveness, encouraging findings have not been replicated, and few papers have been scientifically peer-reviewed."
She summarizes the following methodological weaknesses in the studies that have been done on financial literacy education:
"Despite some resourceful data collection methods, ingenious research designs, and rigorous statistical analysis techniques, studies have been unable to overcome issues affecting data reliability, research design, measure validity, and interpretation of results."
She cites weaknesses in relying strictly on surveys completed by participants in financial literacy programs:
"Research on FLE often relies on surveys (Vitt et al. 2000; Lyons et al. 2006) in which consumers evaluate the education, assess their own knowledge, report their behaviors, reveal their financial condition, or recall their exposure to FLE. But responses to surveys conducted to evaluate FLE are vulnerable to social desirability, demand characteristic, and selective recall biases. Each of these errors is likely to inflate estimates of the efficacy of FLE. Their cumulative effect indicates that relying on survey data to demonstrate the effectiveness of FLE is unsound."
She recognizes the inherent difficulty in measuring outcomes other than participant surveys, including:
"Educators explain that their clients do not want to be tested...Tracking financial behavior is even more difficult. Because we generally do not know who will participate in FLE in the future, academics can view pre-FLE behavior only retrospectively, when the consumer’s memory has faded or has been distorted by ensuing events. Direct observation of consumer decisionmaking after FLE, were it logistically possible, would alter behavior."
She identifies the lack of a control group is the largest methodological shortcoming of program evaluation:
"The biggest methodological problem that undermines the results of FLE efficacy studies has been the lack of adequate controls that would be needed to demonstrate the causal links from education to literacy to financial decisions and behaviors. Typically, academics use as the treatment group consumers who chose to receive FLE. Common sense suggests many unobserved ways in which these consumers differ from those who do not participate. In addition to being better informed or more motivated (Bernanke 2006), those who attend may have more free time for researching and making financial decisions or less embarrassment and denial about having personal finance problems or having made “bad” decisions in the past."
She describes the three step model (financial education begets financial literacy which begets better financial decision-making and behaviors) as a fundamental challenge in determining efficacy of financial education:
"To empirically evaluate the model, researchers must employ measures of the following: exposure to financial education, financial literacy levels, and the quality of financial decisions and behavior. To validate this particular model, they must demonstrate that any link between FLE and improved behavior is moderated by increased literacy. But locating reliable measures of each stage of the model and of the model overall has proven challenging."
She provides a "consensus" view of what is meant by good financial behavior which provides a good template for financial literacy training (bold is mine):
"...where cost-effective, consumers should perform an adequate search for information about alternatives, should expend the needed resources to analyze those alternatives objectively, should base decisions on that analysis by trading off incommensurate costs and benefits where needed, should plan for the future and implement those plans through budgeting where necessary, should select financial products that meet needs without paying excessive prices or incurring excessive risk, should have a personal financial safety net through insurance and/or precautionary savings, and, once a safety net is established, should accept those risks that present a positive probability of higher returns rather than only low-risk low-return alternatives."
Given the measurement difficulties that she lays out in her paper, Willis's solution focuses on creating a norm-based educational approach and a corp of pro-bono counselors to provide customized solutions:
"These academics present an intriguing possibility that it is not financial literacy, but a norm or rule-of-thumb of thrift that mediates between FLE and savings rates. Rather than providing support for the current FLE model, these results suggest an alternative model of norms training leading to changed behaviors...The second public policy model suggested by the research above stems from the studies of credit, retirement, and homeowner counseling. It is plausible that intervention by the counselor and individualized financial advice could improve consumer financial welfare. This raises the possibility that rather than education, a better public policy response to consumer finance problems might be a national system of licensed pro bono financial advisors."
So, what rules of thumb are out there now when it comes to student loan debt levels? Here are just a few I came across:
- Anya Kamenetz (from NY Times): "A good rule of thumb for student borrowers is that your total graduating debt should be less than the expected starting salary in your likely field."
- Mark Kantrowitz: "If you borrow more than twice your starting salary after college, you will be at high risk of default.."
What are some other financial norms that would be helpful to students?