OERu: Developing an Agenda Together Sept 18 - Oct 2, 2013

Quality issues for the OERu

Quality issues for the OERu

by Wayne Mackintosh -
Number of replies: 24

Hi everyone,

Quality assurance and credible credentials are the foundations on which the OERu is being built.

It is not in the interests of OERu partners to engage in practices which could jeopardize their stature or accreditation status. The OERu will not compromise on quality.

At the1st meeting of OERu Founding Anchor partners we confirmed that contributing partners joining the network must be recognised and accredited institutions within their own jurisdictions, but that this was not sufficient. To facilitate cross-credit within the network, anchor partners will aim to quality assure the courses and open education practices that occur between the course and credential and to agree processes for this purpose. True to the remix philosophy of open, we agreed to reuse and adapt existing tools and procedures, most notably the quality frameworks developed by the European Foundation for Quality in e-Learning (EFQUEL) under the OPAL and OERTest initiatives.  We also agreed to approach the Commonwealth of Learning (COL) for guidance and support in adopting and modifying the Transnational Qualifications Framework (TQF) for the OERu which was originally developed for the Virtual University of Small States of the Commonwealth. The OERu will be meeting at COL headquarters on 4 November 2013 to progress work on the TQF.

Questions for our discussion in preparing for the 2nd meeting of anchor partners:

  • How should the OERu partners promote and ensure quality?

  • What are the issues associated with an institution with questionable quality standards joining the OERu?

  • How do quality perceptions of the OERu model impact on standing with regional qualification authorities around the world?

  • What advice can you offer the OERu move the quality agenda forward?

With due recognition to the complexities of quality, looking forward to a productive discussion for the OERu to progress the implementation of agreed quality assurance mechanisms for OERu courses. 

In reply to Wayne Mackintosh

Re: Quality issues for the OERu

by Steve Foerster -

Wayne asks:

"What are the issues associated with an institution with questionable quality standards joining the OERu?"

I'll be particularly interested in the responses to this.  I'm leading a group that's starting a new institution that's making a great deal of use of OERs in our curricula, but as with any new institution, we're not immediately eligible for accreditation in our jurisdiction.  Obviously we're working for that sort of recognition as soon as it's available, but in the meantime despite our stated commitment to providing the best quality education we can, we're not in a position to participate in a number of consortia, including OERu.

In reply to Steve Foerster

Re: Quality issues for the OERu

by Wayne Mackintosh -

Hi Steve,

As an open educator, it's a tough question to answer because restricting membership of the OERu network to a “closed” group of institutions appears to go against the philosophy of openness. This approach seems unfair to:

  • Institutions like yourselves who have a clear commitment to quality but are awaiting formal accreditation status

  • Institutions who may not meet the “quality standards” of the OERu network for reasons outside of their control (eg lack of funding, capability or experience in designing high quality courses.)

As you know, in our model credentials are conferred by the recognised partner institutions, not the OERu network itself. The accreditation status of higher education institutions is typically governed by the respective country's Higher Education Act and our partner institutions are required to operate within these legislative frameworks which lay down the rules associated with the authority to confer degrees.

As part of our philanthropic mission, we release all OERu courses under free cultural works approved licenses as a core principle of engagement and do not require password access to our materials. We will not restrict any institution from access to or use of the courses we develop. In this way we can share back to the wider education community recognising that the courses we build are originally based on OERs developed by the wider open education community. Hopefully in this way we can help build the OER ecosystem for the benefit of all. I would imagine that institutions like yourselves will also pay forward in sharing back OERs you improve. 

In reply to Wayne Mackintosh

Re: Quality issues for the OERu

by Hemlata Chari -

Hello I agree with you Wayne that OER need to be open at the same time there should be regulation to the rigor of the course. At times there are very substandard materials, I feel if  there is a way where we can ask the course developers to rework and submit the quality will be maintained.

I am not an expert like you all, but for our online course I have a team of subject experts who run through the material and if the quality is not maintained we send it back tto course writers, this is a humungus task.

I liked your idea of building OER ecosysytem

Hemlata

 

In reply to Hemlata Chari

Re: Quality issues for the OERu

by Wayne Mackintosh -

Hi Hemlata,

Great to read your contribution and thanks for posting from a non-partner perspective. 

You're absolutely right, quality assurance is essential and is a challenging task. The advantage of our networked OERu model is that we can share the load for quality assurance, thus generating efficiencies which are hard to replicate with single institution models.

I can assure you that the quality thresholds of developing open courses with open design approaches are an order of magnitude higher than closed courses because we can't hide poor quality behind password access ;-).

Another opportunity for the OERu network is to think about "OERu certified" courses where our network specifies the minumum thresholds for quality -- but this is something the partners will need to discuss.  

 

In reply to Wayne Mackintosh

Re: Quality issues for the OERu

by Ken Udas -

I am not sure if this is important to the broader community, but for partner universities accreditation is more than certifying course quality.  It speaks to more broadly defined standards across virtually every aspect of university governance and operations (admissions, student services, academic freedom, retention, etc.).  It is through this accreditation process that institutional (not limited to instructional or academic) standards are certified.

In reply to Ken Udas

Re: Quality issues for the OERu

by Wayne Mackintosh -

Hi Ken,

That's very relevant to our discussions -- the quality issues extend to a broad range of instuttional operations of our partners which the OERu collaboration must incorporate into our deliberations as we find solutions for the future.  

In reply to Hemlata Chari

Re: Quality issues for the OERu

by Haydn Blackey -

Hi Hemlata,

I think you make an important point when saying:

I am not an expert like you all, but for our online course I have a team of subject experts who run through the material and if the quality is not maintained we send it back tto course writers, this is a humungus task.

This is a resource intensive model, and not always scalable. Whereas critical peer review of content is something which can be used on a continuous improvement basis and which partners in the OERu have a vested interest in achieving, as the quality perception of the OERu will be the perception of the quality of its weakest offer, not its strongest.

Cheers,

Haydn

In reply to Wayne Mackintosh

Re: Quality issues for the OERu

by Steve Foerster -

The good news is that since OERu allows those on the outside to participate in these sorts of conversations, I suppose for our purposes there's little practical difference in whether we're officially a member or not.  THat's doubly so in that, like any institution, we can use OERu materials if they meet our students' needs.  And as you say, it's our intention that all material we release publicly will be tagged so that they're of maximum use to everyone, so there's that as well.

In reply to Steve Foerster

Re: Quality issues for the OERu

by Wayne Mackintosh -

Steve wrote:

I suppose for our purposes there's little practical difference in whether we're officially a member or not. 

So this raises a legitimate question for our anchor partners: Apart from building open education ecosystems and social inclusion, what then are the benefits of being part of the OERu network and how does transparent planning contribute to our future success?

I think there are a number of distinct advantages taking into account that once your institution achieves accreditation you will also be free to join our network :-):

  1. While I dislike industrial analogies for education, the OERu network is very similar to the notion of co-opetition where we agree to collaborate on components of our systems in order to “compete” better. Consider for example, the collaboration between Toyota, Peugeot and Citroen who share design, component parts and a jointly owned manufacturing plant to produce competing city cars. In the OERu we get to share “production plant” for the benefit of individual partners, eg central infrastructure for hosting OERu courses plus shared quality assurance models.

  2. Our transparent and open planning systems build trust for prospective partners without compromising our network advantage. Prospective partners can “try before they buy” without the need for a big marketing budget. As a non-profit open collaboration, we don't have hidden business secrets. On the contrary, that's our business secret: the more partners who join, the more effective our network becomes increasing the return on investment for individual members. It's in our interests to openly share our business models and practices. We have a proven and trusted track record of openness dating back to 2006 – we've been in this game for a while. The open internet is a place where its easy to discredit the “brand value” of a collaboration irrespective of the media flavour of the month. The establishment of the OERu network pre-dates the Coursera's, Udacity's, edX's and FutureLearn's of this world.

  3. The major leverage point of the OERu network, is that the individual institutional contribution to open is minimal – restricted to assembling and maintaining two courses. We are nearing the threshold where we can reduce the agreed course contribution while having access to the equivalent of a full set of courses for degree study. That's hard to replicate using single institution models when the OERu network is able to resolve the complex challenges of course articulation. When institutions use our courses, they're obligated to attribute the OERu source - -that's free advertising for us ;-). 

It's smart philanthropy and I look forward to the day when you can announce your own accreditation and consider joining the OERu. At least with the OERu family, you will have both prior and inside knowledge of who you would be dealing with ;-). 

In reply to Wayne Mackintosh

Re: Quality issues for the OERu

by Maria Droujkova -

Wayne, will partner institutions be paid for their participation, out of the money OERu makes to sustain itself? If so, it can be a major point for them, not during the pilot stage, but in the future. The OERu model is infinitely scaleable, and it does not compete with their existing customer base - it's for the next billion, etc. 

In reply to Maria Droujkova

Re: Quality issues for the OERu

by Wayne Mackintosh -

Maria wrote:

Wayne, will partner institutions be paid for their participation, out of the money OERu makes to sustain itself? If so, it can be a major point for them, not during the pilot stage, but in the future. The OERu model is infinitely scaleable, and it does not compete with their existing customer base - it's for the next billion, etc. 

Correct, the OERu model is scalable and fiscally sustainable. This is how it works:

  1. OERu partners pay a nominal membership fee to the OER Foundation which covers the cost of the central shared infrastructure to support free learning. We only need 10 new members for a fiscally self-sustaining model without reliance on 3rd party donor funding.
  2. Recurrent cost for providing summative assessment services is guaranteed because partners provide this on a fee for service basis.
  3. We are not competing with existing institutional markets -- as you say its for the next billion who don't currently have access to affordable tertiary education. 
  4. Staffing is scalable because each partner contributes a 0.2 full-time equivalent staff member to assemble and maintain their course contribution to the network. Most OERu partners are allocating normal development time as part of normal operations, so no new money is required.
  5. The OER Foundation is a non-profit which means all additional revenue must be used for charitable activities. We will be able to allocate surplus revenue for commissioning the paid development of new OERu courses for the benifit of the network. 

It's smart philanthropy -- we can achieve our goals sooner depending on the rate of recruiting the remaing 10 partners. So if any insitutions on the list are interested in joining to become anchor partners (a status membership tier for institutions who join before the official launch on 1 November 2013) -- the should email me offline at wayne@oerfoundation.org.

 

In reply to Wayne Mackintosh

Re: Quality issues for the OERu

by Haydn Blackey -

Hi Wayne,

a great topic for our next session, and one at the heart of lots of debates in Higher Education today. My starting attempt at some answers:

  • How should the OERu partners promote and ensure quality?
    For me there are two roles that come from opposite directions. Firstly we can build as a network on the range and breadth of experiences in the quality frameworks we experience internationally. As an institution assessed by the UK Quality Assurance Agency (QAA) and through the QAA have links with the European HE Quality Framework we are clear about what quality means in our context. But does it mean the same in the context of our partners. In this context the UK Quality Code, though it is perhaps too detailed, is clear about the standards a provider needs to meet. Through the OERu we can compare those standards on an international basis. Secondly in terms of content, rather than learner experience, we can provide strong peer-feedback for the material the OERu offers. While accreditation is dependent on meeting the quality code, the quality of learning offer still matters and I'm sure as we share material that the network will provide guidance, criticism and development opportunity which will improve what we offer.
  • What are the issues associated with an institution with questionable quality standards joining the OERu?
    This is a significant one for us. Given the expectation that the University of South Wales' brand could be harmed by working with partners that don't meet rigorous standards, we would avoid this at all costs. While 'recognition by an appropriate governmental body' is not an absolute assurance of quality - it does act as a cipher for the assurance of quality, so it would be difficult for us if that filter was not in place.
  • How do quality perceptions of the OERu model impact on standing with regional qualification authorities around the world?
    As any OERu provision would be approved and validated by our own institution the question is about our institution systems being measured by the quality code, not by the OERu itself being measured in that way. So this question is less critical for us. However, if the OERu became known for poor quality processes or poor learner experience then institutional review by the QAA might focus more on that. So the better the profile, the easier the assurance process.
  • What advice can you offer the OERu move the quality agenda forward?
    I think it would be useful to build on the experience of many of us here of a wide variety of quality systems and processes to learn from each other, so we have something robust which speaks to the context of all the partners.

 

In reply to Haydn Blackey

Re: Quality issues for the OERu

by Wayne Mackintosh -

Hi Haydn,

You can rest assured that none of the OERu partners will compromise their brand by working with partners that don't meet rigorous standards. The guarantee that the OERu network provides is a powerful one -- namely that OERu partners retain decision-making autonomy over which courses they would consider for local credit transfer.  

Speaking to quality assurance experts, they have advised that we consider a to two tier approach for the OERu which aligns well with your thoughts:

  1. Work with organisations recognised by appropriate governmental body -- which is a necessary requirement, but not sufficient to guarantee quality.

  2. Have the OERu network quality assure courses and processes (which we can scale within the network -- perhaps thinking about OERu "certified" courses & processes.)

As a network, we haven't discussed the detail of what an "OERu certified courses and processes" might mean and I think this is something which the 2nd meeting of partners should begin to discuss drawing on their own expertise.

On one end of the spectrum is how our network should deal with organisations which may not meet generally accepted or agreed quality standards. On the other end of the spectrum is the possible reticence of high quality organisations working in countries which are bombarded with offshore provision from questionable providers in joining our network for fear of what the local QA agency might think of them joining our OERu network. Here the idea of “certified” OERu courses may assist in this regard.

At this stage of our OERu journey all our members are of high standing – but as a network we will need to consider appropriate mechanisms of how we manage this in the future to retain the quality of the “OERu brand”. 

In reply to Wayne Mackintosh

Re: Quality issues for the OERu

by Haydn Blackey -

Hi Wayne,

it is great to see that OERu's aspirations and our institutional aspirations are so well aligned.

Cheers,

Haydn

In reply to Haydn Blackey

Re: Quality issues for the OERu

by Mika Hoffman -

I agree that a strength of the OERu model is that partner institutions issue their own credentials, so if a given course does not meet the standards required for a given institution (or its accreditors), that institution can simply choose not to recognize it for credit. The relevant perceptions of quality, then, are those internal to OERu: members whose materials fall short will need to commit to improve, and each institution will have to continue to implement its own quality measures anyway. That being said, if there is an agreed-upon standard within OERu, it can make for streamlined institutional acceptance: if a given institution knows that OERu standards are like its own except for specific areas that it will need to check, then its accreditors might be OK with a policy that says that the institution will check the offering to ensure that it's in compliance with its unique standards, but won't have to go through an entire QC process each time.

In reply to Mika Hoffman

Re: Quality issues for the OERu

by Christine Horgan -

Mike: eCampusAlberta (eCA) works with something like 16 colleges in Alberta. In addition to an institution's own quality measurements, eCA has its own rubric, too. This helps ensure that there is a common quality in the courses from the different members. It's a process that seems to work. Cheers, Chris Horgan

In reply to Christine Horgan

Re: Quality issues for the OERu

by Mika Hoffman -

Yes, that's great if there's a system-wide agreement on quality standards--and it'd be great if OERu could have such a thing. However, given that the members are all over the world, with different accreditation systems governing them, I don't think it'll be possible for us to have one single set of standards that will satisfy everyone's accreditors. So having one that gives the member institutions a certain level of comfort and predictability should be the goal--and then each institution can identify what else might need to be added to satisfy their accreditors.

In reply to Mika Hoffman

Re: Quality issues for the OERu

by Wayne Mackintosh -

Mika,

I think you're right, I don't think it is possible for us to develop a single set of standards to satisy everyone's needs. We also need to be mindful of ensuring pedagogical neutrality. 

I do think our network will be able to aggree "standarised elements", for example the "optimal" size of a micro course so that it fits the different credit systems which use notional learning hours as oppossed to competency-based frameworks. As you indicated in an earlier post, this will support the scalability of the model.  

In reply to Christine Horgan

Re: Quality issues for the OERu

by Wayne Mackintosh -

Hi Christine,

eCampusAlberta have an impressive set of quality assurance resources. Are any of these resources available under a CC-BY or CC-BY-SA license which the OERu could consider adopting and modifying for our own purposes.

If not, would eCampusAlberta consider relicensing these resources under an open license. All OERu process outputs are available under open license and you are most welcome to use anything we produce.

Working together is more productive than working alone.

In reply to Mika Hoffman

Re: Quality issues for the OERu

by Wayne Mackintosh -

Mika wrote:

I agree that a strength of the OERu model is that partner institutions issue their own credentials, so if a given course does not meet the standards required for a given institution (or its accreditors), that institution can simply choose not to recognize it for credit.

As an aside, there is also a "competitive" element within the network which fuels quality. Individual partners would be concerned if their quality standards are not recognised for credit by another OERu partner. The open development model also contributes to continous cycles of improvement and innovation.  So we work hard at producing the best we can. The experience gained from OERu course development using open design models is ploughed back into institutional design models for their own closed courses.  

In reply to Wayne Mackintosh

Re: Quality issues for the OERu

by Joyce McKnight -

Here's my two cents...Joyce McKnight

 

 

Questions for our discussion in preparing for the 2nd meeting of anchor partners:

 

  • How should the OERu partners promote and ensure quality?  I think that the current standard (i.e. highest accreditation possible in home country) is best for the partners. Others are free to use the materials and/or have staff involved as some sort of affiliates but should not be listed as partners.

  • What are the issues associated with an institution with questionable quality standards joining the OERu?  I believe that those who partner with the organization and/or continue to partner with the organization will want to be assured of the standing of the other partners.  Since at least some of the partners are considered to be "non-traditional" and continually battle to be considered equal to more traditional institutions, the partnership portion of the OERu dare not risk its reputation and, therefore, their reputations.

  • How do quality perceptions of the OERu model impact on standing with regional qualification authorities around the world?  OERu at least at present is probably a miniscule consideration in accreditation which is a major undertaking with many components as someone pointed out in this thread...but even small components can be important.  It is surprising (and sometimes dismaying) what one member or other of an accreditation team will find distressing.   Therefore, it is important that OERu be perceived as a very positive part of the institution's efforts.   In the US at least colleges and universities are supposed to show strength in teaching, research, and community service.  I personally think that OERu's major gift maybe in enabling its members to brag about their service to humanity through the collaboration and perhaps their contribution to research in learning, especially life-long, self-directed learning.   While the course development aspect and sharing resources may be internally important to the partnership, it may actually weaken individual institutions applications for accreditation if only because it will be hard for traditional educators who make up the bulk of accreditation teams to understand.   On the other hand, the idea of reaching millions of currently unserved people is very appealing from the service mission angle...not to mention that it is what I hope we are really about here.

  • What advice can you offer the OERu move the quality agenda forward?  Keep the current rules for participation as partners.  Encourage the partners to offer existing courses (adapted to the openness criteria) so there is no difference between courses offered by the OERu and their own online and campus courses.  Gain a really good reputation for serving students and enabling them to consolidate their learning for transfer...or just to use in their vocations.   Don't try to be all things to all people and don't even seem to be trying to be an educational institution in your own right.

 

In reply to Joyce McKnight

Re: Quality issues for the OERu

by Wayne Mackintosh -

Hi Joyce,

Thanks for your feedback -- which is well aligned with how we are implementing the OERu collaboration.

Since our inception, the OERu consortium has agreed that we must ensure equivalence and parity of esteem for qualifications gained through the OER university network when compared to local credentials.

You're right -- the OERu gives tangible expression to the OERu partner's agendas of social inclusion while maintaining brand and accreditation status. 

In reply to Wayne Mackintosh

Re: Quality issues for the OERu

by John Stampe -
One thought. With all of the different standards already discussed, one needs to be careful with duplication of standards. While some duplication cannot be avoided, duplicate standards increase the amount of work and can complicate things without much value.
In reply to John Stampe

Re: Quality issues for the OERu

by Wayne Mackintosh -

Thanks John, good advice. Working in the OER arena which advocates reuse and remix we should eat our own dog food and avoid the temptation of reinventing old wheels.