Several participants have discussed the importance of building evaluation into the process of instructional design with, at least at this point, more questions than answers. Robbie, Cindy and Sarah discussed student evaluations of a course, and both Cindy and Sarah pointed out the importance of identifying exactly what we mean by evaluation and what is being evaluated. http://scope.lidc.sfu.ca/mod/forum/discuss.php?d=61&parent=294, http://scope.lidc.sfu.ca/mod/forum/discuss.php?d=61&parent=332, http://scope.lidc.sfu.ca/mod/forum/discuss.php?d=61&parent=299 For example, do you want to assess what students have learned and to what extent they have achieved the learning outcomes, or do we want to assess the student?s satisfaction with the course and the instruction they received?
Barb described a turning point in the design process when the learners? evaluations of her course spurred the instructors to make necessary revisions. http://scope.lidc.sfu.ca/mod/forum/discuss.php?d=61&parent=324.
Finally, Sarah said, ?I hope we can focus a bit on how to assess what students have learned and to what extent they have achieved the learning objectives. How do you/we as designers do that?? http://scope.lidc.sfu.ca/mod/forum/discuss.php?d=61&parent=332
Thanks, Vivian, for raising this critical issue for designers up to the surface. I'm looking forward to hearing others' thoughts and experiences, but meanwhile, Marg Riel and her colleagues wrote a chapter for a forthcoming Tim S. Roberts book called, Self, Peer and Group Assessment in E-Learning (If you haven't seen his earlier book, Online Collaborative Learning: Theory and Practice, I highly recommend it). They presented it at AERA 2005. I think anyone interested in this topic will be intrigued. I've attached it as a pdf file.
Sarah,
Thanks for sharing this resource. I struggle with how to go about structuring and supporting peer assessent and your article transforms this question to the level of culture. That is, how do we create a culture where learners are comfortable giving and recieving critiques of thier work to and from fellow learners. It requires an environment where sharing fosters trust and learners are comfortable being open about their experiences.
This is a lovely design goal and one that I'd like to use. Have you or others had experience designing or working with this kind of learning environment?
Thanks for sharing this resource. I struggle with how to go about structuring and supporting peer assessent and your article transforms this question to the level of culture. That is, how do we create a culture where learners are comfortable giving and recieving critiques of thier work to and from fellow learners. It requires an environment where sharing fosters trust and learners are comfortable being open about their experiences.
This is a lovely design goal and one that I'd like to use. Have you or others had experience designing or working with this kind of learning environment?
In response to Vivian's question, Have you or others had experience designing or working with this kind of learning environment?, it brings to mind a mid-term, public self-assessment activity I use in my courses to push the envelope on shifting the course culture from competitive to collaborative.
The idea is to ask students who are in week 6 of a 12-week seminar to go back and review their contributions to date and find one that they consider their most important contribution to the learning of the whole class. The full activity is presented here: http://www.concord.org/courses/facilitating/sample.html.
A few students will have something to post. Others will experience an "ah!ha!" moment and the culture of the class shifts greatly from that point forward.
At the end of the course, I build on that mid-term experience by asking them once again to search back, but this time through the postings of others and to bring forward one posting someone else contributed that greatly supported their learning. I also make a rule that no posts can be presented twice. The result is what I call a course "Gallery" of greatest insights that the learners build themselves.
Sarah
The idea is to ask students who are in week 6 of a 12-week seminar to go back and review their contributions to date and find one that they consider their most important contribution to the learning of the whole class. The full activity is presented here: http://www.concord.org/courses/facilitating/sample.html.
A few students will have something to post. Others will experience an "ah!ha!" moment and the culture of the class shifts greatly from that point forward.
At the end of the course, I build on that mid-term experience by asking them once again to search back, but this time through the postings of others and to bring forward one posting someone else contributed that greatly supported their learning. I also make a rule that no posts can be presented twice. The result is what I call a course "Gallery" of greatest insights that the learners build themselves.
Sarah
Sarah, you've posted such a simply and beautiful activity to foster student self-reflection, self-critique and peer-critique. I especially like it because it seems to be relatively non-threatening to the student, it can be added easily to an existing or new course, and the time commitment for the instructor appears to be very manageable. You have used an assessment activity as the vehicle for collaboration and your students produce a collaborative product - the Gallery.
I've always believed that the more learners take responsibility for their own learning and assessment, the better learners they will become. Could you elaborate on your students' perceptions of these activities and how successful these activities are for you and your students?
I've always believed that the more learners take responsibility for their own learning and assessment, the better learners they will become. Could you elaborate on your students' perceptions of these activities and how successful these activities are for you and your students?
In reply to Sarah Haavind
Re: self-assessment and creating a culture of collaborative exchange
by Cindy Xin -
Sarah, it is a fine example of integrating assessment with learning. I'm a fan of low-stake assessment aiming at learning. By low-stake assessment I mean non-graded and specifically targeted (at a learning goal or objective) kind of assessment that often done by individual students on their own or through their peers. This kind of assessments are agile that can be relatively easily and yet thoughtfully designed. They can be used in the process of teaching of learning for self-monitoring and peer-review, and ultimately fostering self-reflection and critical evaluation.
I have an activity I used in online discussion in the sense described above. The idea is to have a small group of students (e.g., 3-4 members) working together and individually at the same time. The group is signed to work on a central topic of discussion. Each individual member of the group is responsible for a subtopic by building a discussion around it through engaging other members from the group. They are instructed to contribute to and critique each other's subtopic. The interesting thing is that the students quickly figure out that in order to build an intriguing case for their own subtopic, they all have to critically contribute to each other's discussion area while at the same time posting and responding within their own areas. It's a kind of borrowing the "prisoner's dilemma" concept - the best chance of succeeding individually (to "escape") is to actually help out each other and work collaboratively. After a period of discussion, say one or two-weeks, each member of the group writes a summary of issues being critically evaluated in his/her subtopic area and the conclusions (sometimes tentative) reached. The second stage of the activity is to have the group work together to build a case or report on the central topic by linking and integrating the subtopics.
In the above example the individual summaries and the final group report can be graded; however, the discussion process is not - the quality control is done by individual members through peer-review and self-reflection.
I have an activity I used in online discussion in the sense described above. The idea is to have a small group of students (e.g., 3-4 members) working together and individually at the same time. The group is signed to work on a central topic of discussion. Each individual member of the group is responsible for a subtopic by building a discussion around it through engaging other members from the group. They are instructed to contribute to and critique each other's subtopic. The interesting thing is that the students quickly figure out that in order to build an intriguing case for their own subtopic, they all have to critically contribute to each other's discussion area while at the same time posting and responding within their own areas. It's a kind of borrowing the "prisoner's dilemma" concept - the best chance of succeeding individually (to "escape") is to actually help out each other and work collaboratively. After a period of discussion, say one or two-weeks, each member of the group writes a summary of issues being critically evaluated in his/her subtopic area and the conclusions (sometimes tentative) reached. The second stage of the activity is to have the group work together to build a case or report on the central topic by linking and integrating the subtopics.
In the above example the individual summaries and the final group report can be graded; however, the discussion process is not - the quality control is done by individual members through peer-review and self-reflection.
Assessment and evaluation are not my domain of expertise, but I have serious reservations about there being such singular answers. What you measure and how you measure it are going to hinge largely on the topic being taught, the environment where it is happenings (a for credit course versus a training sesssion versus a graduate level seminar), and how the information is to be used (is it to improve the course? is it to provide help/guidance for the student? is it to satisfy institutional assessment agendas?).
We've had eportfolio guru Helen Barrett here recently, and she makes a great distinction between assessment for learning versus assessment of learning.
And while I do think assessment to meet objectives makes sense, it tends to make courses or learning rather one dimensional and isolated- where do you assess the connections learners make outside or to other learning? We still tend to teach in our discipline silos as if someone in chemistry may never need to write an essay or someone in history may never need to do some math. That's how the education system works yet the world does not.
We've had eportfolio guru Helen Barrett here recently, and she makes a great distinction between assessment for learning versus assessment of learning.
And while I do think assessment to meet objectives makes sense, it tends to make courses or learning rather one dimensional and isolated- where do you assess the connections learners make outside or to other learning? We still tend to teach in our discipline silos as if someone in chemistry may never need to write an essay or someone in history may never need to do some math. That's how the education system works yet the world does not.
Alan,
You've opened many questions for me. Could you expand on Helen Barrett's ideas about assessment for learning versus assessment of learning?
And you ask, "...where do you assess the connections learners make outside [the learning objectives] or to other learning? We still tend to teach in our discipline silos as if someone in chemistry may never need to write an essay or someone in history may never need to do some math." This is an excellent question and my first thought is that we need to write learning objectives that reflect these connections, but I have a funny feeling that you wouldn't be satisfied with that. :-)
Thoughts?
You've opened many questions for me. Could you expand on Helen Barrett's ideas about assessment for learning versus assessment of learning?
And you ask, "...where do you assess the connections learners make outside [the learning objectives] or to other learning? We still tend to teach in our discipline silos as if someone in chemistry may never need to write an essay or someone in history may never need to do some math." This is an excellent question and my first thought is that we need to write learning objectives that reflect these connections, but I have a funny feeling that you wouldn't be satisfied with that. :-)
Thoughts?
Vivian,
I cannot do justice to Helen's work, and you will find a wealth of it at her Electronic Portfolios site http://electronicportfolios.org/ such as Conflicting Paradigms and Competing Purposes in Electronic Portfolio Development (PDF).
In my simple minded interpretation, she looks at assessment for learning as the practices set up to enhance the learner's performace-- it includes the reflective practices and "storytelling" aspects that Helen as pushed the last few years in advocation electornic portfolios as a tool for the learner. It is assessment menat primarily FOR the learner.
Assessment of learning seems to be the high stakes assessment institutions look for to be able to give broader information on how well they are doing as an orgranization.
Both are important, but often we seem to be at ends of the spectrum. I would guess a difference is in identifygin who the information is for and what scale it is applied.
I cannot do justice to Helen's work, and you will find a wealth of it at her Electronic Portfolios site http://electronicportfolios.org/ such as Conflicting Paradigms and Competing Purposes in Electronic Portfolio Development (PDF).
In my simple minded interpretation, she looks at assessment for learning as the practices set up to enhance the learner's performace-- it includes the reflective practices and "storytelling" aspects that Helen as pushed the last few years in advocation electornic portfolios as a tool for the learner. It is assessment menat primarily FOR the learner.
Assessment of learning seems to be the high stakes assessment institutions look for to be able to give broader information on how well they are doing as an orgranization.
Both are important, but often we seem to be at ends of the spectrum. I would guess a difference is in identifygin who the information is for and what scale it is applied.
An upcoming CIDER session will be of interest to this group:
Friday December 2, 2005
University of Alberta and Athabasca University - Canada's Open University
Facilitators: Katy Campbell and Rick Kenny
Topic: Instructional design research
Start: 11:00 a.m. Mountain (10:00 a.m. Pacific, 1:00 p.m. Eastern)
CIDER sessions are free, but advance registration is required by emailing Brenda Koritko at bkoritko@sympatico.ca
Friday December 2, 2005
University of Alberta and Athabasca University - Canada's Open University
Facilitators: Katy Campbell and Rick Kenny
Topic: Instructional design research
Start: 11:00 a.m. Mountain (10:00 a.m. Pacific, 1:00 p.m. Eastern)
CIDER sessions are free, but advance registration is required by emailing Brenda Koritko at bkoritko@sympatico.ca