Why do we want to evaluate this? Teachers want and deserve recognition for their efforts in voluntary informal PD. I need to show that my role and the support offered is making a diffference (grant funded project). We need as researchers to better understand where our efforts are best placed.
~ Bron
Currently I create learning that rides the line. I use the formal methods knowing that I would lose learners without them (and thus clients) but I integrate learners' abilities to construct their own learning throughout so that the many problems of formal training do not destroy the flow of learning.
Depending on the group I am working with the context is set with tools to connect and support for collaboration and community building. For instance, I used webcams for new hires who had a full month of homestudy to complete before they even learned about the job they had been hired for. They spent endless hours alone reading. I broke up the time with some elearning and set up study groups (telephone) and study partners (webcams) so that each day they had others to talk to for support, peer to peer learning and then groups came together at the end of each week in a facilitated webinar format to talk about their work together. Before the end of the month I also had them out with experienced employees shadowing them on the job. It helped to explain a lot.
I look for ways to evaluate while knowing that without the informal parts of the program the rest is almost useless. I know this because previous to being involved, the company experienced so much pain with the lack of effectiveness of the homestudy portion of their onboarding process that they looked to redo it. I believe learning processes are very tricky in that learning happens between people and without reflection built in very little happens without others. I am looking for how to qualify the informal interactions as the instigators of learning within the formal system.
I would love to be able to evaluate the informal methods to support further use of them and help learners make better use of them.
Pamela:
Your comments have got me thinking.
Initially, and despite the title of the seminar, I was thinking in terms of personal (vs. professional) learning and so any evaluation would be personal and probably subjective.
Teachers often see evidence of informal learning in the questions learners ask, the reflections and observations in their written responses, and the nature of their musings in discussion groups (in-class and on-line).
Certainly, the broader knowledge of the learner is picked up in the formal evaluations (of assignments). I wonder if informal learning is "informally" evaluated in the professionalism component of a course. If so, I also wonder about, and worry about, the subjective nature of such a mark.
For now, I can see the value of capturing informal learning (perhaps through journaling/blogging), but I'm wondering if we should be trying to evaluate informal learning if it has a grade attached to it.
Cheers, Chris Horgan
I was thinking that evaluation of informal learning would certainly be qualitative...from written comments, interviews and observations. I guess I was also thinking that it would be difficult to separate formal learning outcomes from the informal processes. I agree that graded informal learning evaluation is a difficult mix but I can identify how to think about evaluation of formal learning.
The question about informally evaluating informal learning is particularly interesting. I would love to hear more about that from you and others. What might that look like? Is this an observation, level of participation, or analysis of ongoing comments?
P
"...informally evaluating informal learning...What might that look like? Is this an observation, level of participation, or analysis of ongoing comments?...
-I've seen time logged on - not real I think because who knew what they were doing?
- I've seen number of posts/comments - but how substantive are they
And some people do best ruminating, reading off-line/on-line, thinking about it all and that may be way more meaningful than the 'often poster'.
The prolblem for me is putting a quantitave evaluation on qualitative information - I know it can be done but really like the self report. Perhaps a self report with a 'likert-style' self rating that translates into a mark? (though those can be manufactured too). Perhaps a 'grade' isn't always required. Or a mix of hard quantitative data supplemented by a softer qualitative grade that is instructor assigned from observation of the activity...??
Peer to peer evaluations outside a community context might be interesting to try. Has anyone used peer to peer evaluation?
Re: Informal Learning Contexts - Quest Atlantis
The old line "What is measured, matters" is accurate if you have some idea already about what matters and to what degree. So, (formal or informal) learning - about what, for what purpose, to do what?
I think that we can measure behaviours, change, or difference (or some combination) but to do so you must define what you want to measure and to what standard / criteria. Then the is the consideration of under what conditions does this happen or is appropriate / valid.
best
alicemac
Re: Informal Learning Contexts - Quest Atlantis
I am beginning to get the feeling that evaluating formal and informal learning will ultimately be the same. Learners may be enrolled in formal learning environments but learn informally. I would be interested in measuring the impact of informal learning in formal learning environments. Can formal learning exist without informal learning?
However, before we begin to evaluate anything, shouldn't we know what we are evaluating? So what is informal learning and how is it connected to formal education? Well, according to Cross (n.d.), "Informal learning is learning without borders. Organizations improve it by removing obstacles, seeding communities, increasing bandwidth, encouraging conversation, and growing networks. It’s a natural way to learn and grow" *.
*Forgive me if I don't use formal references.
Re: Informal Learning Contexts - Quest Atlantis
Nellie:
I doubt that formal learning can exist without informal learning. Perhaps informal learning is a bit like the white space around typed words on a page: the space has power because of it ability to shape our interpretation of and response to those very words.
However, some informal learning is incidental (or accidental) learner vs. purposeful, informal learning. As informal learning gets looped up into formal learning, the purposeful learner is going to show that additional learning in his/her work, but it's not going to be easy to measure. I'm also not sure that we should measure informal learning in any formal manner--but I'm willing to be persuaded otherwise.
Please look at my next comment as "electronic thinking aloud." Many learners don't actually know how they learn and yet that knowledge is powerful and valuable information. Some learners will take to journalling and critical reflection like a duck to water....and others will only come to value this learning after being introduced to it --"taught" how to do it, if you will. Perhaps, in addition to the items you mention in your closing comment, informal learning needs to include techniques for making use of informal learning.
Cheers, Chris
Re: Informal Learning Contexts - Quest Atlantis
You said: I'm also not sure that we should measure informal learning in any formal manner--but I'm willing to be persuaded otherwise.
Many stakeholders such as Ministries of Education, teacher's training colleges, career specialists, teachers, curriculum and instructional designers/specialists, program evaluators, school administrators, parents, and students may be interested in knowing how much learning is going on as a result of classwork (formal) and how much outside the classroom (informal) for the purpose of improving instruction and perhaps decreasing the number of teaching hours in one subject and increasing the hours in another. For example, I teach English as foreign language (EFL). Most of my students learn English incidentally or accidentally via TV, the internet, from family members, friends, and from listening to music. It just so happens, there is a shortage of English teachers in Israel right now. Having less hours would resolve the problem of recruiting faculty for specific subjects such as EFL and placing more emphasis on others like Mathematics that cannot be acquired informally.
Perhaps we can list a few formal ways of evaluating learning and see whether any of them can apply to informal learning.
Re: Informal Learning Contexts - Quest Atlantis
Nellie:
I've been giving some considerable thought to what informal learning looks like....and there are, of course, several types.....but "what does it look like in post-secondary courses?" is what's rattling around in my mind right now.
So, again, hiding behind "electronic thinking aloud"....I'm wondering if evaluating informal learning is a task best left to the learner and that if there has to be a formal evaluation why not a learning contract where the learner contracts for a grade based on meeting several goals (which are determined by the learner).
Now, right away, I can start arguing the other side of this item...learning contracts aren't well received in all post-sec institutions, in some schools, by some faculty and administrators....and then there's the learning (formal or otherwise) that's required by faculty and learner to make the best use of a learning contract....still, I wonder if it's something worth exploring.
Chris
Re: Informal Learning Contexts - Quest Atlantis
I would also like to come up with an evaluation plan for self-evaluations and peer feedback. How about compiling a list of criterion?
Nellie
Pamela I really enjoyed reading your post. In Ontario we have enacted the Accessibility for Ontarians with Disabilities Act that mandates training by every business, agency, government etc to ALL their employees to ensure they have awareness in customer service, communications, employment and the built environment releated to disabilities. You can only imagine the scope as virtually every employee in Ontrio begins to receive this training about now - and businesses need to capture and report on this training.
Obviously there needs to be a formal piece to this training (on-line with tracking) but to be truly effective, (as you point out) there needs I think to be some informal, collaborative learning as well. For that we need engagement of learners with eachother and learning objects. I can't think of a way to evaluate this less formal training - perhaps it can only be through self report and annecdotal to get at the real reflection and learning. We can always quantify what the learner can demonstrate at the end of formal training but how can we quantify - or qualify- the actual reflective learning in addition to that more formal piece? or maybe we shouldn't as its very individual to each...
:) susan
I struggle with the use of self report and reflection with training. If you know who your audience and know that they are both used to reflection and self report then it can be used super effectively. Many of the populations I work with are hands-on practices for whom reflection might not be the ideal form. However that is not to say that urging the use of reflection is not valuable. I think corporate training tends to dumb down learning so that it is not perceived as a "time sink" which makes reflection a precious commodity. If learners were more used to using it, though, what a valuable change that would be!
I certainly use anecdotal comments whenever I can get them...I agree with you that they are valuable. Self-reports can be useful but unless the value is there for learners to want to invest the time they can be thin. Again this is a time issue in that learners often think that learning is considered contained by the time in the classroom or on the computer. I have had better luck with leadership training, managers and executives, when it comes to understanding the value of participant feedback. They will often take the time (do they feel more empowered to use their time as they see fit?) to provide a wealth of input. They often are very good at using informal learning to its advantage as well. I think there may be an empowerment issue here now that I think of it.
I was wondering about observation and interviews...hard to get the time for but maybe one way to go?
Pamela
Hughes, Hilary E. (2008). Incidents for reflection in research. In 5th International Lifelong Learning Conference, 16-19 June 2008, Yeppoon, Queensland. Retrieved April 12, 2009, from http://eprints.qut.edu.au
Re: Informal Learning Contexts - Quest Atlantis
I agree with the importance of developing critical reflection. Learners need to have the skills to evaluate what they learn and to write about it; that takes time and practice. I think these are improved by taking the time to write.
Evaluations of self learning can incorporate the reflective process to one's own learning. It would be good if we all took time to ask ourselves the essence of what we are learning. This might help learners (all of us) develop a sense of how to help ourselves and others. The evaluation's benefit -- to what purpose -- perhaps, mainly to improve recognition of what is actually learned and be credited by self and sometimes others.
I recently had the pleasure of participating in Images4education the informal online course equivalent of "Let's put on a show." Check it out!
And of course SCOPE.
Great to "see" you here. You gave an excellent presentation on twitter as part of a ning workshop on integrating technology and teacher mentoring.