EVALUATION AS AN ELEMENT IN GOOD LITERACY PRACTICE
The provision of good literacy practice is the goal of everyone involved in learner-centred community based adult literacy programs. It is well accepted that instruction which recognizes the complex nature of literacy and focuses on the strategies and processes involved in reading, writing and learning is a fundamental ingredient in good literacy practice. What is not so well recognized is the role that evaluation should play in the provision of good literacy practice.
The following story illustrates some of the issues associated with evaluation in community based, adult literacy programs.
It should be apparent from the above story that:
Evaluation tools and procedures must provide meaningful and useful data that can be communicated in ways that satisfy the needs of everyone involved in supporting, delivering and receiving literacy provision. Such tools are as important to good literacy practice as is good instruction.
In fact it can be argued that good evaluation is more important. Without good assessment tools and procedures, there can be no real foundation on which to plan the appropriate instruction required for good practice. In any case, what is certain is that without good evaluation and good instruction there can not be good practice.
Careful consideration of the following 5 questions should help clarify some of the factors that need to be considered when implementing an effective evaluation plan for learner-centred, community-based adult literacy programs. These questions are:
The learner, the instructor, the program managers, the community supporters and the program funders all need evaluation information.
Colleges, other training institutions or programs, potential employers, government agencies and the media are some examples of others who may want information about the program and/or the learners' achievements.
Learners want to know whether or not they are achieving their goals. They need to be able to celebrate their successes and develop confidence in themselves as learners. They have to know what they have already achieved so they can make decisions about setting new goals and planning the next stages of their learning.
Instructors need to know where instruction should start and what direction it should take. They need to know whether or not their planning and instruction has been effective. This information will enable them to make decisions about adjusting their instructional plans, setting new goals and/or selecting different materials or approaches. Finally, they also need to be able to celebrate their good practice and develop confidence in themselves as diagnostic and prescriptive teachers.
Program managers, supporters and funders need to know if the program is providing good literacy provision. They will want to know if learners are achieving their goals. They will also need to know if the program is fulfilling its mandate and meeting its objectives.
This kind of information will enable them to make decisions about program delivery and management, as well as about whether or not they should continue to support the program. They also need to be able to celebrate the positive results of their support and be reassured that their time, effort and financial contributions have made a difference.
Colleges, other "outside" institutions, and employers will want to know if students meet their requirements. They will need to know that the evaluation data provided by the program is reliable. Only then can they be confident of a smooth transition from literacy program to their institution or workplace.
Even though stakeholders in the literacy program have their individual reasons and purposes for wanting evaluation data, the following reasons are common to all of them:
In summary, it can be said that for everyone the goal of evaluation is to allow for the examination of past performance and the initiation of future action.
A study of current evaluation research suggests that the only way to ensure that the above goals of evaluation are achieved is to choose evaluation tools and procedures that will provide meaningful and useful data. To clarify what is meant by meaningful and useful data, it is helpful to define these words as they relate to evaluation and then look at specific examples that illustrate the collection of this kind of data.
The following examples illustrate how evaluation data can relate directly to objectives:
If a learner's objective was to improve his comprehension of text by using self monitoring strategies, having the learner tape himself thinking aloud while reading would be an effective evaluation procedure. The tape would provide meaningful data because it would reveal exactly what he is and is not doing in relation to his goal.
If a teacher's instructional objective was to teach writing using a process approach, ongoing evaluation of students' writing at various stages in the writing process would provide her with meaningful data. The samples of work in progress would provide concrete evidence of the strategies and techniques students were using to arrive at a finished piece of writing. Such evidence would be meaningful to the teacher because it would explicitly reflect where instruction had been effective for students and where it had not been effective.
Some researchers suggest that if no action will result from the data collected through a particular evaluation tool or procedure then the evaluation should not be carried out.
Celebrating progress and good practice, setting new learning goals, rethinking instructional objectives and strategies, seeking out specific materials, inviting an expert into the classroom, renewing a commitment to support the program are all examples of actions that could follow from useful evaluation information.
In order to collect evaluation data that tells both where you are and where you need to go in relation to your objectives, careful consideration must be given to the selection of evaluation tools and procedures.
Only those evaluation tools and procedures that are aligned with both the broad goals and the specific learning, instructional and management objectives of the program will provide data that can reveal what progress has been made in achieving these goals and objectives and what future actions need to be taken.
It is widely accepted that in learner-centred, community-based adult literacy programs, good literacy practice occurs when the goals, objectives and delivery of literacy provision mesh with the philosophy that defines this kind of programming.
If evaluation tools and procedures are selected on the basis of their alignment with instructional practices and the program goals and objectives, then they will be congruent with the beliefs about the nature of literacy and literacy learning that make up this philosophy. Evaluation will then be woven together with philosophy, objectives and instruction and play an integral part in good literacy provision.
Good literacy provision depends on the alignment of instruction and evaluation with philosophy. To select appropriate evaluational and instructional tools and procedures, we need to have a clear understanding of the basic beliefs that define learner-centred, community-based, adult literacy programming.
These beliefs and the objectives, methodologies, and evaluation procedures that follow from them can best be understood by considering each of the following three questions:
An example of a specific instructional objective following from these beliefs would be the decision to teach the pre-reading strategies of skimming, scanning, activating previous knowledge and making predictions. An evaluation procedure that is congruent with this objective would be one that allows the learner the time and opportunity to use these strategies with a variety of texts and in a variety of situations. It would also be one that provides the learner with a way of documenting and reporting on his/her use of the strategies.
Examples of evaluation tools that are congruent with these beliefs and objectives are:
When evaluation tools are congruent with beliefs and objectives they provide data that is meaningful and useful to both the instructor and the learner.
In contrast, a timed reading assignment that required students to demonstrate their understanding of the text by answering multiple choice questions on the content, would not be congruent with this instructor's objective or with his/her beliefs about what learners need to know and be able to do.
Such an evaluation tool would not provide any explicit information about the learner's use of prereading strategies. Not only would it not provide any data related to the objective but it would also inhibit the use of the very strategies that the instructor was trying to promote.
Misaligned evaluation tools not only fail to provide useful data, but they can also have a very negative impact on learners. They can, for example, result in students feeling confused and frustrated. They can also cause learners to lose confidence in themselves as learners, in the instructor as a reliable and helpful teacher and even in the program as a worthwhile place to spend their time and energy.
All results that are inconsistent with beliefs that the program holds about the need for learners to know how it feels to have a positive attitude to learning and to themselves as learners.
What is valued in literacy instruction?
According to the current philosophy of learner-centred, community-based adult literacy programming, valued literacy instruction is instruction that is participatory and interactive.
Such instruction is directed by the individual learner's goals, shaped by his or her learning style, needs and interests and driven by his or her learning pace. It builds on all facets of the learner; his or her strengths and weaknesses, previous knowledge and experience, and motivation.
Valued literacy instruction is instruction that engages the learner in a broad range of authentic tasks presented in real situations and involving "real" people.
Valued instruction is instruction that recognizes the multi-dimensional nature of literacy learning and emphasizes the processes and metacognitive aspects involved in reading and writing rather than on the mere production of finished pieces.
Above all, valued literacy instruction is instruction that empowers learners by fostering independence, self-esteem and positive attitudes towards learning.
Looking at an evaluation tool that is not congruent with the objectives and methodologies that will follow from these beliefs will best serve to highlight what appropriate tools and procedures would be like.
Formal standardized tests would not be an appropriate choice for learners and instructors who are setting goals and engaging in the kind of practices that would be aligned with these beliefs about valuable literacy instruction.
Why are these inappropriate evaluation tools? They are inappropriate because:
Formal standardized tests are so misaligned with the instructional objectives and methodologies that originate in the philosophy of learner-centred, community-based, adult literacy provision that they disrupt and counteract good literacy provision and result not in meaningful and useful assessment information but rather in lost instructional time, confused students, and frustrated teachers.
In contrast to formal standardized tests, appropriate evaluation tools and procedures would:
Interviews, conferences, journals, observations, and portfolios are all examples of appropriate evaluation tools.
These tools would provide individualized information that reflects the dynamic, functional, multi-dimensional nature of literacy. They would also promote the instructional objective of learner empowerment.
As learners identify and discuss the knowledge and skills they have acquired, as they reflect on what they have achieved and what they need to do next, as they analyze and evaluate their work, they will be assuming ownership of their learning and thus be empowered.
How can literacy learning best be demonstrated?
Since the philosophy of learner-centred, community-based, adult literacy programming accepts the belief that literacy is functional, then the skills that have been learned and the conceptual understandings that have been developed need to be demonstrated in authentic situations for real purposes.
Observations of learners reading writing, speaking and listening while engaged in genuine tasks for real purposes would be an appropriate evaluation procedure.
Since this philosophy also accepts the belief that literacy is multi-dimensional, then an appropriate evaluation process must provide opportunities for the learner to demonstrate a broad range of performances and behaviours.
Using a variety of evaluation tools and procedures would allow for the collection of this kind of data.
Furthermore, since this philosophy accepts that literacy learning is a process of growth and change that occurs over time, then an appropriate evaluation procedure must allow the learner to demonstrate that growth and change has occurred over the time that he/she has been in the program. Using portfolios is an example of this kind of evaluation procedure.
In summary, meaningful and useful evaluation information can be obtained if evaluation tools and procedures are:
If the evaluation process meets these criteria then it will enhance instruction and provide information that can evaluate progress as well as initiate future actions.
Knowing what evaluation tools and procedures are appropriate to use is one of the indicators of good literacy practice. Knowing when to use them is another.
The practitioners in learner-centred, community-based, adult, literacy programs generally recognize the need to have procedures in place to gather initial and final assessment information. However, the need to have a plan to collect evaluation data on a regular, ongoing basis is not as widely accepted. Yet regular, ongoing evaluation is critical to good literacy practice. Why? Because regular, ongoing evaluation:
Evaluation can be a key element in good literacy provision if practitioners implement an evaluation plan that: