I think a feedback poll for each reading assignment could help gauge the helpfulness of content for readers of various experience. Additionally, it can help the facilitators weed out less useful content and scaffold weak areas.
The feedback mechanism would be simple: http://thestaticfrost.posterous.com/injecting-a-feedback-mechanism-into-the-learn
In terms of analytics, one thing I'd like to know is how many of my total posts got responses, and how many responses they got.
Then I'd wanna mine down into which posts got responses, what kinds of responses they were, and whether they furthered the overall discussion.
I'd like this too. I would add some form of tagging users.
Don't get me wrong, I do not mean to make a statistical or automatic profile of users, but you could add custom tags to each user ... Of course, to identify users whom you connect to the system you described is needed first.
Me gustaría esto también. Yo le agregaría alguna forma de taggear a los usuarios.
Que nos se confundan, no me refiero a realizar un perfil estadístico o automático, sino que uno pudiese agregar tags personalizados a cada usuario... Por supuesto que para identificar a los usuarios con los que más te conectas necesitarías primero el sistema que describiste.
- how valid an idea it was I put forward in a post, based on (critical) feedback
- how relevant the post was to the conversation, based on # of responses
- how much of what I was thinking resonated with others
- what kinds of communications work best in a forum context (ie. is provocation more effective than diplomacy in generating responses)
- who else in the forum got what I was saying and took it further
- who could offer thoughtful alternative perspectives
In terms of tangible influence, I think I'd see a discernible (though I can't say how I'd measure it) change in my communication style in discussion forums.
Las preguntas propuestas son:
What would you like to show?
Composition: the components that pieces make to a whole.
Distribution: How one variable's values are distributed.
Comparison: How do values change over time or in relation to one another?
Relationship: How one or more variables changes as a direct function of another variable.
Not sure yet.
De ahí se pueden identificar diferentes casos de uso.
As an educator for the classification categories exemplify the use of the concept. Descriptive statistics for example if you are exemplifying the search for statistical graphing and chart types is the following use classification.
The proposed questions are:
What Would You like to show?
Composition: That the components to make a whole pieces.
Distribution: How one variable's values Are Distributed.
Comparison: How do values change over time or in relation to One Another?
Relationship: How Changes one or more variables as a direct function of Another variable.
Not sure yet.
From there you can identify different use cases.
You did say to dream ;)
the representations you list and this URL which
maps the representation of TV stories over time and the component parts.
BBC's "The Mythology Engine - representing stories on the web":
I also miss the "I like" buttom that Facebook has. I guess I'm too accustom to "liking" posts instead of actually replying to them. But it's been a good exercise. Facebook has made me a little lazy :O
Kami Mutan wrote,
This is possible here in the forum using Marginalia. This quick video shows how to make basic annotations (this includes tags) to posts. A second video shows more advanced annotation features. What I like most about Marginalia is that it is so flexible, and wide open for different uses.
I am definitely missing the option of being able to custom tag posts
I've been thinking about the best way to annotate forum posts in this course in order to generate a summary documents. So far I've only tagged a few posts containing suggestions or issues related to participation in the course ('user experience' and 'course considerations'). So I see one category as being course administration and facilitation. A second category would be around emerging themes and recurring questions. Another possibly the suggested tools.... endless!
Thanks for sharing the link to the quick video that shows how to make basic annotations. The presenter was suggesting the annotations could be used to take notes. The annotations might be a useful way of coding comments. The annotations might contribute to knowledge building of the group if they are made public. An individual might use them to track her thinking, whilst exploring the resources.
This being my first MOOC, I am somewhat surprised by the volume of messages generated. I created a filter in my inbox and direct messages to a LAK folder; but 2 days in I find that I am mostly skimming, unless of course I want to respond to something. I've also found that even though I might nit want to respond to someone's post, what they write is so compeling that I'd like to connect with them on LinkedIn or Academia.edu for future discussions.
My blog is http://kamiresearching.blogspot.com (it's in Spanish)
Was thinking that LinkedIn could provide one model of a PLE-like space into which one could pull (learning) content, contacts, discussions, feeds, etc -- all ripe for analytics tools.
Inside of Class analytics:
- Number of views for my posts
- Top ten most viewed posts (other than the instructors :D)
- Country and City Location of those who view and comment on my posts (to view geographic closeness and identify cluster areas)
- Length of reader stay on my post
- Maybe something like the adaptive database that Amazon or Barnes & Nobles online, where you get an automatic prompt like, "People who liked this post also liked these".
- A way to see whether any link I posted was clicked on and visited.
- Some form of performance assessment/ credibility ranking as far as quality of posts
- At a glance graphic of participant active useage based on MST :D So I can see when most of us are on
- A buzz indicator (something that shows what the most popular instructor and participant posts are).
- Class statistics: average length of stay in site, which components were viewed, number of posts, etc.
Top ten most viewed posts (other than the instructors :D)
Country and City Location of those who view and comment on my posts (to view geographic closeness and identify cluster areas)
Length of reader stay on my post
Maybe something like the adaptive database that Amazon or Barnes & Nobles online, where you get an automatic prompt like, "People who liked this post also liked these".
:) GREAT IDEA!!
A way to see whether any link I posted was clicked on and visited.
Some form of performance assessment/credibility ranking as far as quality of posts
DISAGREE, I think that's too subjective. What is important for me is not important for everyone here. Maybe the statistics of the user will suffice.
At a glance graphic of participant active usage based on MST :D So I can see when most of us are on
WHAT IS MST?
A buzz indicator (something that shows what the most popular instructor and participant posts are).
AGREE, just to go against the current and visit the least popular posts XD
Class statistics: average length of stay in site, which components were viewed, number of posts, etc.
It would be great if that statistics could be linked to our personal sites, blogs, etc. I mean to have an exact idea of how many people signed and visited other people's blogs, made facebook friends, etc. A way to measure the online connection generated by he course :)
Excuse my English
I would also like to see some sort of "priority inbox" like google that takes data from forum posts and prioritizes then for the learner
While I agree with your comment about the subjectiveness for the performance measure. I was thinking about some metric that would appeal to the competitive nature of everyone and measure their contribution to the course. You could have something as simple as a "like" button common in social media or a start system that readers could assign as they felt like it. :D
One of the things I've noticed about the MOOC setup is the feeling of being lost with so many other voices. As a participant, I'd like to see who is driving the discussion and be able to identify them quickly and easily.
As an instructor, I'd love to see confirmation of who my class influencers are and the peer groups that get established within a course.
I understand both of your points regarding the 'metric' for quality/credibility
And if we can personalize it... Who's more credible for me... like the Hunch results were supposed to be I'd vote for it too.
The same for the instructor... different students will be influential in different ways so different criteria will turn different leaders.
I liked the way you framed the conversation.
Like you, I believe that it would be difficult to develop a performance assessment or credibility rating for determining the quality of a post. People who are enrolled in the course are diverse and we have very different reasons for studying analytics and for engaging in a MOOC.
I am not sure how to track class statistics...
Class statistics: Any effort to track class statistics would be 'partial'. Perhaps it would be possible to see how long someone took to type a response, but what does that really mean? A course and learning management system can track the time participants spend on the site. Semester after semester my students personalize their learning. Final papers, presentations and group projects reveal little correlation between time spent in the shell and actual learning. Since the LAK11 course is distributed, thinking, reading, etc. will occur both on and offline and in multiple sites. Wouldn't it be nearly impossible to actually track and individual's activity?
It might be interesting to try out different rating options in the forums in this course. As an example, in the open for anything forum I added the Separate and Connected ways of knowing scale. You can use the drop-down menu for each forum post to flag that contribution. (I hope I've flicked all the right switches so everyone can use this tool -- Let me know!)
This is just an example. I can set up scales here in Moodle with whatever words we'd like to use. We can also implement a rating system (one star, two stars, etc) although I'm not sure if that kind of feedback is as valuable as using language. For example, maybe a rating that indicate how individuals relate to the post would be useful:
1. need clarification
2. helpful now
3. advanced, need to revisit later
I'm making this up as I go along so I hope it's making sense!
What are your ideas for scales to use in this course?
The voting up or down and summarizing features, that can be created to appear as a kind of intro to the thread, would also be really helpful for people to be able to scan the forums with greater prioritization and filtering. The voting is a little like the FB "I Like" but tracks the voters and raises the level of the post answers in the order visible. The summary feature is a great way to kind of keep an ongoing editable abstract of the posts.
Those two analytic features might serve as great ways to reduce the kind of MOOC fatigue and overwhelmed feeling that so many people can suffer from after about the third or fourth week. There might be some added complexity with the threaded nature of discussion forums, but I know I would appreciate something like that a whole lot in any of the MOOCs I have experienced.
Fred Haas wrote,
While I agree with you that this may help, how does the system deal with people who play it to their advantage?
The voting up or down and summarizing features, that can be created to appear as a kind of intro to the thread, would also be really helpful for people to be able to scan the forums with greater prioritization and filtering. The voting is a little like the FB "I Like" but tracks the voters and raises the level of the post answers in the order visible.
The communication and discussion show us how the platforms use for
E- learning, need to change, we live in word where we learn to just (like) something, (vote) and follow the post with the most comments. The E-learning platforms should evolve to. I am not an analytics fan but I'm learning so much, with all your points of view. I may twitter this :-)
Apostolos Koutropoulos wrote,
This statement really bounced out at me. I wonder about this with applications like twitter which show number of followers. A decision to follow might be based on how popular that individual appears to be, rather than the quality of what they heave to offer, or, more importantly, now connected you might be to their ideas so that you can build on them.
The one thing that I would be cautious about is the group think that might develop from only following popular threads.
I spend a lot of time communicating online and meet (and seek out) new people all the time. I notice that some of the most thought-provoking dialogue can be with individuals who are finding their way in a new domain, asking questions, challenging ideas that others skip over because they're so used to hearing them.
I do see value in somehow showcasing individuals who contribute to and advance collaborative dialogue. But as Apostolos suggests it might have a negative impact if we begin to use popularity ratings as an indicator that those are the conversations we should be paying attention to.
So I'm reading, I should probably change the habit of reading, as I read every post from my email and not from the forum. If I read the email I will not count in the statistics but I suppose that "my life will be more visible" if read in the forum, right?
Good point - i.e. reading online vs. reading in the forum. So, yes, your life is more visible if you read it on the moodle forum. I don't think moodle offers a widget yet that allows email tracking the way that services like Constant Contact do.
As others have stated in this discussion - analytics are only as good as our data. It's a good reminder that all data related to learning analytics is somewhat incomplete - especially if confined to learning that occurs in a classroom or an LMS. We need multi-silo data access. But, that then raises Total Information Awareness concerns: http://en.wikipedia.org/wiki/Information_Awareness_Office
Agreed. This is also related to the "lurkers" (I prefer the label "auditors") of courses, listservs, presentations. They are there, participating in their own way, yet not captured adequately or even accurately in the data.
Is there a possibility that we are confusing data/information, data management, and metrics with analytics?
Personally, I would be interested in tagging trends and how "connected" a bit of information is (does it include links to outside sources other networks?) so that I could also expand my range of connections. As a way of considering how much this course is benefiting me and whether I am using resources effectively, I'd want to understand where the buckets of information are and how I'm actually tapping into those buckets to achieve something...so I'd want to tie the analytics to my own learning contract (after having identified goals I want to accomplish).
The interests should/could be mapped with their semantic relationships, and with the size of node representing numbers of participants with that specific interest. Nodes could be zoomed in and out. (Zooming in to the most detailed level would reveal usernames of participants listing that interest, and if we're dreaming - hotlinks to their presence on the web - blogs, twitter, fb, linkedIn, etc.)
This would be helpful for a number of reasons:
1) finding a subgroup with similar interests/emphasis in this MOOC
2) viewing the breadth and proportions of interests represented by LAK participants
3) It would be most enlightening to see how (or if?) this interest/emphasis map evolved over time. An animation of such would be very insightful, especially if somehow it could be tracked before and after LAK11 in Banff, to measure impact the course, and then the conference had on interests.
Not sure if I think it would be useful for either a learner/user or facilitator or administrator at this point in terms of asking questions:
Tracking code + widgets e.g.
||What technologies are they using
Where they come from
What they do when they are here
Where do they go next
Why do they return to something
Why do they leave something
Do they provide any info about what they are interested in
Do they provide any info about what they are looking for
Do they provide any info about who else they might be interacting with (see below)
What problems might they be having
Why they start something and stop halfway through
Use of translation
Any other accessibility issues
|Twitter, FB, Google Friend Connect,
Additional analytic apps for the above
Disqus, IntenseDebate, Backtype
||Needs to be linked to referrals i.e.
Did they come via a recommendation from someone
Do they reply to a particular person/people
Do they not reply to particular person/people
Do they increase / decrease their number of contributions over time
Where are exit points in a discussion and who leaves
Who else are they already connected to / social tools already using
Are likes/dislikes/votes/rates about particular topics / people
Are there any signs of people being influenced by others existing likes/votes etc
Changes in style, language, tone in comments over a discussion, over period of time
What is tagged and where
- Who would benefit from viewing this information?
- Who can already view this information?
- What is a successful experience?
- Are people are bored/satisfied/ecstatic about their experience?
Some (partially) unknowns
- How much is private messaging being used?
- Why anonymous users are choosing to be anonymous?
(Sorry for always insisting on developing better tools).
Let's take these three items of the social behaviour (see attached image):
- tags (I call topics)
- likes, dislikes, rates, votes (I call UDRIVE)
- who else is online at same time (I propose virtual presence)
can't we have the three items above in each Forum (perhaps only one Forum is then needed).
When adding and entry to a Forum it would be very easy to tag it as belonging to one or more topics, for instance EDM, Privacy, K12 or even Spanish.
Any user should add a topic if he feels the need (or the Forum manager can set a predefined set of topics if he wishes to).
NOTE 1: The operation to choose to which topics an Entry belong must be easy like using checkboxes!
NOTE 2: The filtering of entries by topics should be available!
NOTE 3: If you are only seeing entries of certain topics it should be seamless to make our reply also belong to those topics (enabling, of course, to add or remove any of those topics)!
Any Forum Entry can be marked by readers for Instance as UDRIVE (Uread, Deleted, Read, Interesting, Very Interesting, Exceptional).
NOTE 4: FIltering by the marks we gave should also be possible.
NOTE 5: If all this information, likes and topics, can be repesented in various forms in table and charts then we could have insight on what is going on.
WHO ELSE IS ONLINE AT THE SAME TIME
Virtual Worlds are very good at it. If the Forum has a 3D representation then we should see other avatars that are close to it.
All the above is already integrated in my www.umniverse.com system and soon you will be able to test it if you want (I join an image to open up your appetite).
My main message is: tools should get better! Integrated, easier to use and with instant analytics (which for common users I call them Statistics ;-) ).
There should also be an option to view as maybe a cloud or maybe something else - a tagcloud of related tags to that tag which are clickable or have previews as above.
I'll take a look at other tag systems, namely those that you mentioned, and improve Umniverse Forum object accordingly.
Here I join another view of current graphics/analytical possibilities hoping it meets some of your ideas.
Wishing you the very best with umniverse development,
Umniverse development will continue and it gained a lot from our LAK11 experience.
Let's see how learning can be improved with Virtual Worlds and analytics.
All the best for you,