Best Metholdogy??

Best Metholdogy??

by Terry Anderson -
Number of replies: 15
Methodologies provide the tools and guidance for their effective employment.

Methodologies employed reflect and affect the mind set of the researcher and the research process.

I think most of us would agree, that no one tool set is the best choice to use in every context and to understand every problem. Nonetheless there may be methodologies that are more (or less) in line with those of a funding source or a political meme.

Should a Pan Canadian Research Agenda recommend (or reflect):
  • a particular research approach?
  • a radically 'neutral' stance?
  • or just play down specific approaches claiming value in maximum diversity in the approach recommended.

Your best methodology?
In reply to Terry Anderson

Re: Best Metholdogy??

by Valerie Irvine -
I think the methodology need not be determined - or else be done in a flow-chart manner or venn-overlap manner so it reflects various approaches, and thus, levels of evidence.

I'd like to refer to an initiative by a group in the teacher ed field who tried to establish a distributive research agenda:
http://distr-collab-teacher-ed-research.wikispaces.com/bibliography
I'm linking directly to the bibliography which provides some references for collaborative or multi-institutional research. This might be referenced for this discussion.

I think it would be important to keep a category or tag on each reference or piece of work as I know the government reps are very heavy on the systematic review/levels of evidence approach (along the lines of cochran or campbell - as opposed to a lit review done in a systematic way...). Now, we need not take this as prescriptive for what we're doing per se, but it might be important to at least track what's done along these lines, so it's easier to respond to questions from them.

So, is there an interest in trying to collect data across multiple sites in Canada? or create questions that are answered by multiple co-Is? or co-PIs? in different settings?

On the notion of topics - I'm actually somewhat interested in e-learning acceptance/adoption. I know we have a lot of studies that examine the question of learning or how learners learn or communities of learners interact, but when I'm often pulled into senior management meetings where the question on the table is "why is our uptake so low?" - I'm curious about the determinants. I also teach education students and there are many ways in which technology, when used as a tool, can improve learning and/or admin, so why not the uptake? I know the diffusion of innovation literature, but I'm looking for something that can be testable and used in larger institutional settings, such as the UTAUT model (Venkatesh). I know the lit of similar theories have been tested hundreds of times so as to establish a deepened understanding of these determinants so areas of intervention can be identified. Anyhow, that's one of my current topics.

Valerie
PS - I hope I'm not coming across too one-sided in the discussion. I am very much in favour of mixed methods research and believe this is stronger than any methodology on either side of the spectrum. That said, I know only one side will get the attention of governments... feel free to demonstrate otherwise as perhaps my perception is wrong as I am still relatively new to the field.

In reply to Valerie Irvine

Re: Best Metholdogy??

by Terry Anderson -
Thanks very much Valerie for the link to the wikispace set up by the Distributed, Collaborative Research Model for Technology in Teacher Education project. The wiki is a good example of the type of work I hope we will accomplish in week 3 of this conference. As Valerie noted the references on the last page of this wiki are also useful and seem to provide theoretical insight into some of our earlier problems (as noted by Stephen Downes) with multiple site, Canadian e-learning research projects. For example they list:

Corley, E. A., Boardman, P. C., & Bozeman B. (2006). Design and the management of multi-institutional research collaborations: Theoretical implications from two case studies Research Policy, 35(7), 975-993.
The paper presents a theoretical framework for collaborative models for diverse institutions. It proposes that the success of the inter-institutional research collaboration is to large extent dependent on two factors: the level of development of epistemic norms within the disciplines represented in the collaboration and the type and level of development of the organizational structure of the collaboration. The analysis of two cases shows that the success of inter-institutional collaborations needs a relatively high level of development in either the epistemic or the organizational domain.

They define epistemic norms as, pertaining " to the internal workings of research communities, particularly norms and practices for research, research agenda-setting, incentives and rewards."
We are obvious a very disparate group in the e-learning research community(s) and thus likely do not share a great deal of epistemic norms and our collaborative organizational models are not that well defined either. However, our success at forging a country from many nations may hold some promise. Obviously we need a network and not a tight group organizational model to provide coherence and strength in the midst of out heterogeneousness.

The most extensive discussion in the wikispace noted above seems to concern the breadth of a collaborative research project- Should a relatively few (one?) research project be undertaken that would allow for combining/comparing results from many contexts - or should as wide a range of questions as possible be addressed. Seems this group opted for the narrower approach.

Much food for thought!
Terry
In reply to Terry Anderson

Re: Best Metholdogy??

by Valerie Irvine -
Hi Terry,

I think we need to have a tight organizational model - but this could be in two phases perhaps or two levels. One project to roll out across various contexts would probably be highly valued because of it's completeness. That said, if we can get pan-Canadian commitment to an "accord" of sorts -then we can decide to work together to chose that first question/project in year 1, then next one in year 2, or some fashion. If someone's interest isn't captured in year 1, then we might still get their commitment if there is a promise that their interest is captured in a later year. We need commitment per province of various potential "torch holders" so if one falls out, then another can fill the spot.

I like your idea of the sample email to provincial education reps. However, I think it's important to be very systematic in this - how do we know which ministers are being contacted in which province/territory? it would be good to at least have a point of contact to map out who is being brought in so as to ensure that no one is left out. Perhaps a new thread to just simply post the names of people that have been invited from govt or other organizations. If one wants to be discreet about it, they could fire it off to one of the moderators for posting (sorry!). Thoughts on that? or save the systematic approach to the funding approach I'm about to describe...

To be honest, I like the idea of a funding proposal that we can come together on - so we can all sign off on - and each provincial group of co-PIs agree to submit the complete document to their provincial contacts - ministry of ed or advanced ed or ?? - this document would be the complete pan-Canadian research agenda - including a) funding to support the network and b) funding to complete the research on i) its part in a pan-Canadian study (like the teacher ed wiki mentioned above), and perhaps ii) a provincially-focused study or study on a different question. (?) Each provincial/territory group of researchers would submit the proposal within the same time frame. Let me ask you - if you were a Minister of Ed/Advanced Ed/etc. and you received such a proposal - showing that this group got their act together and had all of these academic players pulled together and you were asked to contribute your 1/13 to the pot, wouldn't you feel inspired to do so? and perhaps even concerned about it being 12/13 funded and your province was the one left out?? Perhaps the Federal or other organizations (Industry Canada, etc.) would be the ones to help contribute to the communication/network/travel funding part or to help cover shortages in a given province/territory.

I've got a new skookum research lab here starting up now (with co-director, Allyson Hadwin). It's a CFI-funded Technology Integration and Evaluation Research Lab worth over $700,000 and could use some of the VC infrastructure to help support/connect with the others across Canada. I know there are many sites out there, but we could easily get funding for the small desktop units, or software, for those who don't have easy or cheap access. Or else I'd love to invite as many of you out here to Victoria to enjoy the daffodils and ocean view in the middle of winter... ;) (speaking of which, we have a wonderful conference centre connected to the Empress Hotel downtown which would be great for another CNIE conference...)

For TOPIC thread: The activity in conducting this research as a pan-Canadian network will be a research project worthy of attention in and of itself - perhaps that should be listed as another topic. The e-research element as an off-shoot - who better to understand the ethics and implications than DE researchers?

For a new STRATEGY thread: Now, it looks like our thread has sort of evolved out of "best methodology" into "best strategy"?? should the organizational/practical elements of how to do this be a new thread? We could repost or move some of these over.

This brings up another point. I know that many educational technology faculty members across Canada have a responsibility to serve the teacher education programs at their universities (as many are posted in Faculties of Education), so it may be that not all attention is focused on the e-learning part of it necessarily, though they may have it as a secondary research interest or be well-practiced in teaching in an online or blended manner. I think it would be important to approach these folks at all PSEs across Canada as they could help to hold up this network and even help to do some of the provincial data collection if it is agreed upon to do a one project/many sites idea. Again, we would want depth per province/territory in case someone puts down the torch, another needs to follow. I think personally that it would be important that these folks be in the status to be eligible for tri-council funding, so they could continue the "attack" on that front. Targeted email invitations would be important.

For the TOPIC thread: For the research component - sorry - should be writing over in that thread... - oh, well - perhaps people could quote/reply in the appropriate thread instead as I'm on a roll... - we should have the primary discipline studies/topics (by DE researchers), but also secondary discipline studies - to serve the stakeholders in other fields (nursing, health, etc.) that may be doing their dedicated programs online or in blended mode. So, this is more about some of the "special populations" being served and perhaps this includes international students specifically. These research streams might allow for additional funding targets to be included - funded by their parent funding agencies outside of education.

Sorry - I've got my eyes on the money piece a bit too much...
I definitely have an organizational bent - gets me carried away sometimes... I'll try to dig out your diagram, Terry, and try to map this onto it. Where did I see it? In your PowerPoint? if you recall, I can find it. The levels of evidence, which I think is necessary, would be the layers underneath each topic bubble.

Curious to hear your thoughts on this - or maybe I'm too ahead of myself in which case hold it as "food for thought."

TTYL,

Valerie
skype: swirvine



In reply to Terry Anderson

Re: Best Metholdogy??

by Michael Hotrum -
I agree with the second and third choice - the radically neutral stance seems to incorporate the last suggestion. I think we have to be in the forefront of redefining what went before us - to accomodate the differences that lie ahead. For example - the use of wikis and blogging as a tool for research development, the sue of collaborative spaces, open sharing of research in process,  the need to use blogged discussions as examples of produsage development, the need to question the design and delivery paradigms now represented by formal learning institutions, the need to expand our offerings and our services beyond schooled events - to capture the informal learning, the connections that are made in the learning process, the need for students to take more direct control of the design and execution of their learning process - these and other "different" actions require that we look at future research efforts as different than past practices. So yes - a radically neutral stance is necessary.
In reply to Terry Anderson

Re: Best Metholdogy??

by Caroline Park -
Somehow screwed up times and missed the presentation but have just listened to the archived version.

I believe that a PanCanadian Strategy starts with the long term goal and builds backward getting wider and wider, to include very diverse, yet linked topics in all possible methods.

This type of planning leads to visual graphics that can be instantly understood.

Then as proposals go forward to funding bodies each researcher or network of researchers can show where they fit into the big picture.

Practice disciplines can include elearning research but they might link in in multiple places depending on the sector of the field.

Caroline
In reply to Caroline Park

Re: Best Metholdogy??

by Madhumita Bhattacharya -
I like Caroline's idea of the visual graphics of research plan and methodologies. A research agenda (or a topic) could involve multiple methodologies. New approaches could also emerge as different fields (areas/subjects) gets interconnected. Diverse approaches with explanation of each and every node and link of the visual map will provide an overview of various methods and their interplay in a distributed environment.
In reply to Terry Anderson

Re: Best Metholdogy??

by Haydn Blackey -
Before moving into Learning as my academic discipline I came from the Marketing discipline. As such I'm tempted to ask Terry who the audience for the research is. This will matter a great deal to any research agenda.

If it is for those who what to know, explore and understand then an approach that is neutral or even discriminates towards the forms of research which currently hold less credence in the public mind would be a strong position.

If however it is for funding and for industry/Government then we need to be able to research and articulate in a language that is understood in their context.

Thus the answer might be different for each audience - unless we are willing to take the risk, as you hinted at yesterday, of establishing the agenda so clearly that we go to Government/industry with our principles agreed and try and change their perceptions.

Trying to do this is fraught with risks. In the UK, and I guess in any society influenced by the scientific culture of the last three centuries of Western society, the primacy of positivism with its emphasis on proof as repeatability and ability to count is so pervasive that politicians, journalists, academic managers and society in general find it hard to value or evaluate research which is not based on this approach. Therefore for a research agenda to challenge these views will involve challanging embedded cultural norms. So a group of interested people like us might have seen the light and be ready to talk in terms of neutral approaches to research, but in an unneutral context we may need to challenge existing approaches before getting our voice heard to the widest audience.
In reply to Haydn Blackey

Re: Best Metholdogy??

by Karen E. Smith -
Haydn's comment about background experience influencing your thinking is interesting. I think that we can all see some "marketing" attached to any enactment of innovation that we would use. Our centre for this marketing shows up in different ways. Four colleagues and I in a US/Canada study about adopting innnovation in teacher education all reported different means of having the change virus within our independent institutions. We measured how this took place through an instructional design model and set each of our change processes together, side by side, on the table. We were unable to report how one institution had to drop their changes and lost one of their most innovative DE programs because change caused too much disruption and eventually things fell apart. Eventually our research group ended up reporting the success and semi-success stories at a conference but it was difficult to go beyond this.
In the end marketing affected how we were able to report the results of our study. We haven't been able to publish this yet because the experiment fell apart.
In reply to Terry Anderson

Re: Best Metholdogy??

by Debra Hoven -
In consideration of the points above, there is certainly the place for research that fits with the expectations and agendas of the intended audience (e.g. government) and certainly we should have clear models and processes for tackling such projects. However, As Caroline paints the picture, it is our long term goals that should be shaping (not determining or directing!) what we research and how we go about it.
With the collaborative consciousness we are creating through this series of discussions now, hopefully we will be able to keep the ground shifting with dynamic aggregation and re-configuration, while at the same time maintaining nodes of critical mass in areas of on-going importance and interest to ourselves as well as to bureaucratic/administrative/policy bodies such as schools and governments.
In reply to Terry Anderson

Re: Best Metholdogy??

by Caroline Park -
As has been indicated, there are several agendas relating to our strategy, depending upon the target stakeholder group. If we work with one stakeholder and develop the steps toward the goal, we will probably be able to tweak the strategy for another stakeholder.

In my mind it is like a tree with the goal at the top (which really has to relate to Terry's introductory comment):

“E-Learning has been recognized by many nations as being critical not only for existing formal education systems, but more importantly for lifelong learning, professional development, re-training of immigrants and creating opportunity those who are deprived of access to traditional forms of education and learning”(Anderson, 2008).

What is the critical thing? Is it access? efficiency?

To get to the critical thing we need steps A,B & C and all need funding, perhaps from different pots, but if we are really good, from a giant e-learning pot that gets created because we are so clear in the mission.

Caroline
In reply to Caroline Park

Re: Best Metholdogy??

by Caroline Park -
I just had another thought while reading George's post on "reduce reuse...." Our strategy would show what has already been done in each branch , leading to what else needs to be done. So we will know where the concentrations of E-learning research are already (reuse?) and presumably the experts in that area and then we REDUCE duplication of effort.

Caroline
In reply to Terry Anderson

Re: Best Metholdogy??

by Kelly Edmonds -

This is a tough question. Taking this on, means taking on traditions in the field of research - of which I am not adverse to. This is how we evolve.

I don't think creating a meta-method has any advantage, as mentioned by some of you. This would become to grand-theorist like, and would contradict newly embraced notions of collaboration, networks, and Web 2.0 attitudes.

However, I suspect it would be beneficial to ensure that whatever direction is taken (I am still thinking on this), it should be backed by sound theory. There has to be a foundation on which strategies, techniques and methods rest, otherwise the new direction will have a looseness, and perhaps a poor finish.

Taking on the metaphor of Web 2.0, there is room for many ideas and methods, but there should be a few underlying paradigms. I am not afraid of paradigms as they will shift in time. And, if I draw on the Web 2.0 notion and theories of interdependence and social construction, the challenge will be to lay this paradigm over traditional methods. And again, I wonder if this is leaning towards a meta-method.

But in my mind, we need a cohesive approach to research methods in order to move forward. If 'anything goes' is sufficient that I think we will end up with a borage of information and ideas, as we have now.

The challenge will be to create harmony, connection and understanding among the research field in order to take e-learning and its research to another level. I advocate to do this theoretically.

Kelly

In reply to Kelly Edmonds

Re: Best Metholdogy??

by Terry Anderson -
Thanks for comments and suggestions on the complex issue of best methodlogies. Perhaps the topic should be expanded to perspectives, as it is a larger description of the world view that guides our methdological choices. Wikipedia lists a number of "perspectives" in their entry on e-learning as follows:
  • instructional design - the traditional pedagogy of instruction which is curriculum focused, and is developed by a centralized educating group or a single teacher.
  • social-constructivist - this pedagogy is particularly well afforded by the use of discussion forums, blogs, wiki and on-line collaborative activities. It is a collaborative approach that opens educational content creation to a wider group including the students themselves.
  • Laurillard's Conversational Model[5] is also particularly relevant to eLearning, and Gilly Salmon's Five-Stage Model is a pedagogical approach to the use of discussion boards [6].
  • Cognitive perspective focuses on the cognitive processes involved in learning as well as how the brain works.[7]
  • Emotional perspective focuses on the emotional aspects of learning, like motivation, engagement, fun, etc.[8]
  • Behavioural perspective focuses on the skills and behavioural outcomes of the learning process. Role-playing and application to on-the-job settings.[9]
  • Contextual perspective focuses on the environmental and social aspects which can stimulate learning. Interaction with other people, collaborative discovery and the importance of peer support as well as pressure.[10]
I like the idea of graphically noting the interesctions and insights provided by all of these approaches and agree that we probably need to have room in our agenda for many perspectives, while highlighting those we feel are of particular value.

Terry
In reply to Terry Anderson

Re: Best Metholdogy??

by Kelly Edmonds -

Thanks Terry and Glen for adding multiple perspectives and emerging theories to this discussion. I sense you are both on the right track in acknowledging multiplicity, and looking at how to work with this. I wonder if Chaos Theory or something along that line would help?

In thinking a bit more on what direction e-learning research could take I found myself wondering why so much research focuses on improvement. I just finished teaching a graduate class on student assessment. The literature is seeped with notions of betterment, improving and strenghtening. That is, teaching, leading and motivating students to do better and better and better. This is a very indiviudualistic attitude that can be seen in the North American society. Purposes of improvement are also seen in research, and the goals for study by others.

Looking at the dynamic nature of the web, produsers, etc. I don't get the sense this evolution and explosion of web uses is about improvement and betterment. I get more connecting, sharing and thinking. Whereas, improvement connotates competition. I think elearning research should explore the experiences of participants, and uncovering their real needs might be more useful.

I think it might be time to let the Enlightenment period rest.

Kelly

In reply to Terry Anderson

Re: Best Metholdogy??

by Glen Gatin -
A few ideas for discussion purposes.

There may be a need to develop a new mix of methods depending on the real questions being asked.

If part of the issue is resistance to the adoption of eLearning and the barriers are cultural then an qualitative approach seems indicated. The questions are "why" questions.

If research is to energize change, any action needs to be directed by theory. Existing theories appear to be lacking. Collaborative application of a Grounded Theory method could be used to develop explanatory theory from a wide variety of data sources, qualitative and quantitative. Emergent theory could be validated by quantitative analysis. Once validated the theory could be used as a basis for action.

Because of the speed of the development of ICT, any research will have to be able to be "rapidly deployed" to be relevant in any prescriptive way. Social networking tools could be used to gather data, facilitate analysis and disseminate results in a cost effective manner.