Anxiety is also at play in certain workplaces, with some fearful of the “risk of showing that a
poor training decision was made by a senior level of management”
(Bélanger and Robitaille
2008: 65). Other common constraints relate to costs, lack of personnel, limited time and
insufficient knowledge or training to carry out an evaluation (Dunberry, 2006 10-11; Bélanger
and Robitaille, 2008: 64-65). Thus, employers in Australia worry about the administrative
burdens of evaluating and reporting. In the UK, the main current barriers to evaluation have
been identified as: securing “buy-in” to the process from managers/staff, the difficulty of the
process, limited time and cost (Pye and Hattam, 2008: 46-47).
Over the years, evaluation of workplace literacy programs has followed the Kirkpatrick model, which remains the most popular and, according to recent studies, appropriate and “adaptable” approach (Pye and Hattam, 2008: 68-70; Dunberry and Péchard 2007: 8, 13). Kirkpatrick’s four levels of evaluation include: 1. Learner reaction to the training program, 2. Learning or knowledge acquisition, 3. Learning or knowledge transfer (job performance), and 4. impact on the business (productivity). As Gray pointed out, employers tend to view training within “a narrow organisational perception of the importance of skill development . . . their primary concern being to achieve returns from their investment in the form of improved performance and cost savings” (Gray 2006: 12). However, program evaluations have tended to revolve around the first two levels, focussed on the learner and learning, with some reporting on Level 3 and very little on Level 4, which is “typically not addressed” (Gray and Sutton, 2007: 52).
Evaluation at Level Four is not easy “because it can be extraordinarily difficult to isolate the
effects of the training alone on certain business metrics, [especially since] employers often
implement more than one improvement strategy at a time”
(CAEL 2006: 105-6). In Canada, for
example, a 2005 report by the Conference Board indicated that only 12% of member
organizations evaluated the business impact of their training programs (Bailey, 2007: 23).
Growing concern about getting employers more interested and involved in literacy and essential
skill training is now helping to fuel a call for a shift in evaluation focus from concentrating on
the learning process and learner gains towards performance and productivity evaluation.
One example of this development is provided by the WoLLNET Project (jointly funded by the
European Commission's Leonardo da Vinci Sub-Programme, the London Development Agency
and the Learning and Skills Council London Region), which is currently “developing a web-based, user-friendly, theoretically grounded Toolkit to enable employers, providers and unions in
participating countries to systematically evaluate the impact of workplace basic skills training
programmes”
. One of the key innovations featured in the project is to “extend the traditional
areas of workplace basic skills training evaluation beyond the typical focus on learner response
and the value to the learner, by measuring the impact of training on individual and organisational
performance”
. The stated goal of this shift is to generate evidence that can help “make a strong
business case”
to employers for basic skills training (WoLLNET Workplace Literacy Language
and Numeracy Evaluation Toolkit Project: http://www.wollnet.org/inbrief_en.htm). Also in the
UK, the Enhancing 'Skills for Life' Project has been tracking outcomes to both workers
(improved LLN skills) and their employers (productivity) (Gray 2006: 74).