Week 2: Rise of “Big Data” and Data Scientists

I don't have any ready answers, but I think this is a really important issue. As with students, the process would need to be transparent and include some types of benchmarks for various levels of "performance".

Analytics could potentially help the overall quality of content and instructional interaction if used in this way. Too many times I have joined an online course where an instructor felt they could simply receive student data inputs without providing content or feedback themselves. As an instructor and student I would find this valuable simply in allowing myself to compare my work with other instructors and perhaps discovering areas in which I could improve. As an administrator I would hope to ascertain information that would help me target specific instructors as potential mentors and mentees in teaching online.

This will be interesting to explore both the potential uses and abuses.

Kae, A number of (I think most) online universities already have instructor performance analytics and acceptable minima are already written into their contracts. Data can be used to assure accrediting authorities that instructors are indeed available and responding. Response times are analysed. Instructor availability, number of inputs, assessment turnaround times, interactions with each individual student and more are collected into assorted charts and used for various QA and HR purposes. The one thing that I have never seen, however, is a serious analysis of the value-added. By that, I mean how much the instructors efforts increase the student's performance. I have seen plenty of raw scores of instructor performance and student Likert-scale perceptions of instructor performance but nothing that closes the loop where the instructor can say, 'Well, if their own effort to use what they had been given had been increased a bit, they might have got full marks." Without that loop closure, feeding back the instructor data into instructional/learning design requirements - especially requirements from people who do not teach the course in that specific target market - is always going to have a large element of guesswork attached.

This would be great in the corporate environment. I would love to get rid of "Happy Sheets" as they are called (Was the room temperature comfortable? Was your instructor prepared?). This was an example of useless busy work that took on a life for itself. The time and infrastructure expended to collect meaningless surveys about student classroom experiences was immensely frustrating. I would like to see a more robust data-driven methodology for analyzing (and therefore developing) instructors.

Good questions! I wonder if there is any tools for analyzing these very helpful points. SNAPP seems only work for analyzing interactions between participants.

Su-Tuan Lulee

This is a great thread, and what I'm interested in as well. While "Learner Analytics" are a great idea to help students modify their behavior (or at least understand what the predicted outcome is), I'm interested in how we can use Analytics to design environments that improve learning outcomes for all students - or those targeted, such as those at-risk.

We have the most potential control/influence on the design of courses and strategies, so it seems like Analytics would be a great tool to help us evaluate which designs work, and which don't, for selected purposes.

Is anyone aware of literature around the use of Analytics to evaluate course and programs? I haven't found any myself.

We're planning this kind of work in the California State University system as an outgrowth of auditing/reporting metrics (such as those previously mentioned, around time on task, last login, etc.).