Week 1: Introduction to Learning and Knowledge Analytics

Use cases for learning analytics in the corporate world

Picture of Bert De Coutere
Use cases for learning analytics in the corporate world
by Bert De Coutere - Sunday, 16 January 2011, 6:54 AM
To make the concepts of learning analytics speak out more, I thought it would we a good idea to list some use cases of how we might apply it in a corporate context. This includes samples of what some companies are already doing or planning to do, as well as to boldly dream where no corporate training has gone before...

Any sample use cases you can think of? (Ill list some to start the discussion.)
Picture of Bert De Coutere
SNA and leadership training
by Bert De Coutere - Sunday, 16 January 2011, 7:00 AM
The use case:
A young manager quarterly runs a tool to picture his social network, as part of his continuous leadership development. Data mining has shown that successful leaders in his corporation typically have 1/3 of their network amongst subordinates, 1/3 with other managers on the same level, and 1/3 higher up the chain. The tool allows the young manager to see the evolution in his network and get some action points to move it to the desired state.

The analytics behind:
Based on data visualization and Social Network Analysis tools. These tools can list a person't network and intensity of its connections by mining digital contacts via IM, mail, tweets, blogs, ...

(btw I just sucked out the 1/3, 1/3 and 1/3 out of my left thumb.)
Fun with Stencils
Re: SNA and leadership training
by Apostolos Koutropoulos - Tuesday, 18 January 2011, 9:05 AM
Interesting scenario.
The one thing I wanted to add here is that it's not always about the position/level of the people you are connected to but a more pertinent descriptor would be "what do these people know?"

It's good to be attune to the beat of the various levels of management and what goes on in the front lines, but often times people's job descriptions don't accurately reflect the skills that they have and how those could be utilized.
Picture of Bert De Coutere
Reverse engineering of learning interventions
by Bert De Coutere - Sunday, 16 January 2011, 7:04 AM
Use case:
I want to become proficient at learning analytics. So I go to the learning portal and ask for competence building activities (I hate to reduce to only 'learning') for a maximum of 5 hours (that's the time I have this week), based on my preferences. (I'm for example intellectually challenged reading academic papers, I prefer slideshare or prezi presentations or video testimonials). The system reverse engineers a personal list of suggested activities. If you insist, you can call it a course or curriculum.

Analytics behind:
Instead of an expert setting the order of activities you need to go through, we use data mining and performance statistics to reverse engineer a successful curriculum. It is based on what people before me have done to become good at X, and filtered for the time I have and historic data on how I learn best.
Picture of Dianne Rees
Re: Reverse engineering of learning interventions
by Dianne Rees - Sunday, 16 January 2011, 8:43 AM
I think the idea of an "intelligent curriculum" based on both competency and interest is a fascinating one. It's also been called adaptive learning technology and Knewton is an example of a company that seems to be using this kind of approach, though I haven't seen many examples of it working for learning that involves more nuanced problem solving. (Knewton mainly focuses on prep for standardized tests).

But your idea of going beyond what the individual is good or not so good at to consider what others have done to be successful is a really interesting one. (And perhaps a system also could consider common mistakes by novices compared to experts). I wonder how we could implement these approaches with the authoring systems and LMS's we currently have (could involve human curation of suggested activities as well...e.g., by course/program alumni?)
Picture of Gillian Palmer
Re: Reverse engineering of learning interventions
by Gillian Palmer - Monday, 17 January 2011, 3:57 AM
Bert, thank you for adding the 'use case' approach here and, yes, I also the idea of reverse engineering successful learning based on individual history, time availability and goals. As learning is becoming more personalised, I suggest that we may need to reverse engineer a number of curricula to arrive at a suggested pathway and we shall also need a 'current conditions' reality check (especially in compliance training). I ahve only jsut downloaded VUE. Do you know if you can overlay different VUE maps and extract the common elements?
Picture of Inge Ignatia de Waard
Re: Reverse engineering of learning interventions
by Inge Ignatia de Waard - Monday, 24 January 2011, 12:08 AM
A great idea to work with use cases. And it would definitely help with self-regulated learning.
Picture of Bert De Coutere
by Bert De Coutere - Wednesday, 26 January 2011, 3:39 AM
OK, I'll try one again. Does anyone else feel comfortable adding some use cases as we go along in this course?

Use case:
Every so many months, I let a tool mine my historic learning behavior. All learning interactions I've done (which were all voluntary and part of a large choice of options I have at my disposal) have left trails like the info I looked up, the blog posts I've done, the happy sheets for training, the experts I contacted, the elearning I completed, the tests done, the coaching given or received, etc. As I'm a self-directed learner, this tool gives me a MIRROR for my own learning behavior to reflect upon. How have I learned best? This is not based on some mythical (un)existing or (ir)relevant learning styles, but based on past behavior, the satisfaction and the impact from that.

Analytics behind:
Keeping statistics on most frequently liked types, best working types, etc etc. Going further, this might be linked to suggestions, but just as a 'mirror' for a learner it is already valuable.

It's an aid in learners to find out how they advance best, away from all guessing and biases.
Picture of Tanya Elias
Use Case
by Tanya Elias - Saturday, 29 January 2011, 12:42 PM
Use case:

Developing training for Customer Care representatives.
Level 1: Baseline performance stats are gathered: New curriculum is developed, agents train, are tested and begin working - same stats are again compared to see how much the new training improved performance. Gives an overall picture of whether new training was effective for the curriculm development team.

Level 2: Baseline stats are gathered. Parts of training and assessments question performance and types of assessment and tied to specific baselline stats. Same stats are remeasured. Gives a more detailed picture of which parts of training were effective for the currciulum development team (i.e., Did learners who scored well on the written assessment questions perform better on the job? or even If learners scored well on question one, did that help them do better on the job?)

Level 3: bits tied together in such a way that at the end of each shift, agent performance is measured and their areas of strengths and weaknesses that day are presented to them. Based on these measures, specific training bits are suggested directly to the agent. Once completed their performance is again measured an presented to them. Curriculum development team gets a report of which lessons are most improving agent performance which they can then use to determine which aspects of lesson design are most important.

Picture of Bert De Coutere
Re: Use Case
by Bert De Coutere - Sunday, 30 January 2011, 12:46 AM
I like very much how this case talks about baselines and stretches into 'the work', and not just into 'the learning'.
Your third level makes me realize I have been focusing too much on making everything automatic, but then we would miss the essential human element. So indeed, get the real baseline information to the development teams for example. A lot of what learning professionals now base their actions on is 'professional intuition' that can be improved with statistically sound baselines and performance data.
Picture of Inge Ignatia de Waard
Re: Use cases for learning analytics in the corporate world
by Inge Ignatia de Waard - Wednesday, 26 January 2011, 5:03 AM
Giving my two cents

Use case, let's say for health care workers
Health care workers need to find the latest relevant health information on specific topics. Most of the knowledge is screened by WHO (world health organization or other health expert institutes), however it is hard to stay on top of those topics that are not put into health guidelines. This pushes those professionals in screening for the 'best' solutions themselves.
It would be interesting if all the networks and places these health care workers visit, are screened and mapped, in order to see which network is used the most of getting to relevant information. And additionally to see whether there are certain health care workers that come up more frequently than others on having correct health information within a certain medical field. If so, you could locate top experts (that possibly do not have a formal title for their knowledge capital) and gather them in a 'knowledge center'. Making it easier for the other health care workers to find relevant information constructed by information production of these resources and people.

analytics behind
Keeping tracks of digital movements, time in reading resources (thinking that it takes more time to read something of interest, then something which is skimmed for possible interesting information, making the first option more relevant to pinpoint knowledge quality).

It would be an aid on finding relevant, field-specific information to develop professional knowledge.
Picture of Sarah Haavind
Would this fit with the health care example?
by Sarah Haavind - Thursday, 27 January 2011, 10:09 AM

I got intrigued by Inge's health care example, which has more universal applications, I believe, and then I saw this: http://scimaps.org/maps/map/inspire_map_107/

What do you think of an inspire sort of tool, could it be useful in your world? Or does it help to clarify how visualization and background analytics could open up more knowledge to be usable?
Picture of Inge Ignatia de Waard
Re: Would this fit with the health care example?
by Inge Ignatia de Waard - Saturday, 29 January 2011, 4:09 AM
hi Sarah, this software surely seems useful as it also might indicate upcoming topics. I am going to play around for it and see what it gives. Thx for the link!
Picture of Bert De Coutere
Re: Use cases for learning analytics in the corporate world
by Bert De Coutere - Sunday, 30 January 2011, 12:49 AM
This is indeed an intriguing example that can be generalised to any topic. There is quite some software out there to extract 'trends' from what is going on in forums and communities. This 'buzz' can feed in to knowledge centres.
Or even more simple example: the new training nuggets on application X or rollout Y can be based on the trends detection in the support forums and help desk calls that are mined.
Picture of Bert De Coutere
Subscribe to performance indicators
by Bert De Coutere - Sunday, 30 January 2011, 12:52 AM
Use case:
This year, I want to improve my presentation skills for example. I subscribe to indicators of how good my presentations are. All the feedback I get in evaluation sheets after a public talk, all the press, all the tweets and blogs about it are analysed and as such I regularly get a status on how I'm doing. I use that to steer my learning efforts (and why not get some automated tips).

Analytics behind:
Mining for performance indications, much like companies are now mining the internet for 'good' or 'bad' about their brand.
Picture of Bert De Coutere
Sales training determined by clients (is this an evil example?)
by Bert De Coutere - Sunday, 30 January 2011, 12:56 AM
This might be an 'evil' use case.
In sales, everything is about the needs of the client, right?

Use case:
Mr X is a sales person responsible for a particular sector and some large accounts in that sector. Every time those clients visit the company's web site, their behavior is logged. If they for example are searching or visiting pages on new products, this goes into the 'TO LEARN' bucket of the sales person, so he knows what to prepare for on his next visit.

Less evil is to do the same for segments like regions. If in some region there seems to be more interest in Y, you can direct your sales attention and sales knowledge/learning on that.

Analytics behind:
Collect the 'triggers' for specific interests, and feed that into the learning path of sales people.
Picture of Hans de Zwart
Portfolio management through monitoring search terms
by Hans de Zwart - Sunday, 13 February 2011, 9:07 AM
You are responsible for the project management portfolio learning portfolio. In the past you mostly worried about "closing skill gaps" through making sure there were enough courses on the topic. In recent years you have switched to making sure the community is healthy and you have switched from developing "just in case" learning intervention towards "just in time" learning interventions. One thing that really helps you in doing your work is the weekly trending questions/topics/problems list you get in your mailbox. It is an ever-changing list of things that have been discussed and searched for recently in the project management space. It wasn't until you saw this dashboard that you noticed a sharp increase in demand for information about privacy laws in China. Because of it you were able to create a document with some relevant links that you now show as a recommended result when people search for privacy and China.
Picture of Bert De Coutere
Re: Portfolio management through monitoring search terms
by Bert De Coutere - Saturday, 26 February 2011, 8:02 AM
I like your concept of the 'what's hot this week' list. In one of the last sessions in this course, I saw examples of how text (in English) can be mined for the kind of conversation that is going on, so I guess text mining can find out what are topics that are/not solved etc...

Just think of the agility and time-to-competence to respond to weekly changing topics!
Picture of Hans de Zwart
Social Contextualization of Content
by Hans de Zwart - Sunday, 13 February 2011, 9:09 AM
Whenever you look at any piece of content in your company (e.g. a video on the internal YouTube, an office document from a SharePoint site or news article on the intranet), you will not only see the content itself, but you will also see which other people in the company have seen that content, what tags they gave it, which passages they highlighted or annotated and what rating they gave the piece of content. There are easy ways for you to manage which "social context" you want to see. You can limit it to the people in your direct team, in your personal network or to the experts (either as defined by you or by an algorithm). You love the "aggregated highlights view" where you can see a heat map overlay of the important passages of a document. Another great feature is how you can play back chronologically who looked at each URL (seeing how it spread through the organization).
Picture of Bert De Coutere
Re: Social Contextualization of Content
by Bert De Coutere - Saturday, 26 February 2011, 8:07 AM
You see this already in a lot of web applications, but not in so many corporate interanets yet. Most of that is for consumers, so I like your suggestion of the play back to see who saw or even used it. (in SlideShare for example you can see where your piece of content is embedded/used/consumed. )

We need to provide not only goodies such as tags and ratings for the consumer of this content, but also provide some feedback and evidence of use of it by others. Thay way people that contribute a video, a blog post, a knowledge base article, a quiz, a checklist, etc get acknowledgement and an overview where it is used.
Picture of Hans de Zwart
Data enabled meetings
by Hans de Zwart - Sunday, 13 February 2011, 9:10 AM
Just before you go into a meeting you open the invite. Below the title of the meeting and the location you see the list of participants of the meeting. Next to each participant you see which other people in your network they have met with before and which people in your network they have emailed with and how recent those engagements have been. This gives you more context for the meeting. You don't have to ask the vendor anymore whether your company is already using their product in some other part of the business. The list also jogs your memory: often you vaguely remember speaking to somebody but cannot seem to remember when you spoke and what you spoke about. This tools also gives you easy access to notes on and recordings of past conversations.
Picture of Hans de Zwart
Automatic "getting-to-know-yous"
by Hans de Zwart - Sunday, 13 February 2011, 9:11 AM
About once a week you get an invite created by "The Connector". It invites you to get to know a person that you haven't met before and always picks a convenient time to do it. Each time you and the other invitee accept one of these invites you are both surprised that you have never met before as you operate with similar stakeholders, work in similar topics or have similar challenges. In your settings you have given your preference for face to face meetings, so "The Connector" does not bother you with those video-conferencing sessions that other people seem to like so much.
Picture of Bert De Coutere
Re: Automatic "getting-to-know-yous"
by Bert De Coutere - Saturday, 26 February 2011, 8:08 AM
Blind dating smile
Picture of Hans de Zwart
"Train me now!"
by Hans de Zwart - Sunday, 13 February 2011, 9:14 AM
You are in the lobby of the head office waiting for your appointment to arrive. She has just texted you that she will be 10 minutes late as she has been delayed by the traffic. You open the "Train me now!" app and tell it you have 8 minutes to spare. The app looks at the required training that is coming up for you, at the expiration dates of your certificates and at your current projects and interests. It also looks at the most popular pieces of learning content in the company and checks to see if any of your peers have recommended something to you (actually it also sees if they have recommended it to somebody else, because the algorithm has learned that this is a useful signal too), it eliminates anything that is longer than 8 minutes, anything that you have looked at before (and haven't marked as something that could be shown again to you) and anything from a content provider that is on your blacklist. This all happens in a fraction of a second after which it presents you with a shortlist of videos for you to watch. The fact that you chose the second pick instead of the first is of course something that will get fed back into the system to make an even better recommendation next time.
Picture of Bert De Coutere
"Signals", but for work not education
by Bert De Coutere - Saturday, 26 February 2011, 8:15 AM
In this course we looked at the Signals project, that shows students if they are green/organge/red positioned to get their grade.

So a signals system for corporate should do the same for your performance. If your sales figures are getting off target, if your project contribution or appreciation is slipping, if your programming code has more bugs, etc you get a warning on your performance and automatic suggestions for learning. The success of this system will depend on it being totally automated and only between an objective 'system' and yourself. Noone gets notified of these preemptive warnings, no managers get any alerts yet... The system is there for you to take action.
Picture of John Fritz
Re: "Signals", but for work not education
by John Fritz - Saturday, 26 February 2011, 12:56 PM
"The success of [Signals for work] will depend on it being totally automated and only between an objective 'system' and yourself. Noone gets notified of these preemptive warnings, no managers get any alerts yet... The system is there for you to take action."


I sure hope this doesn't come across as petty or sniping, but Signals is not automated. Students only see a red, yellow or green stoplight if a Purdue instructor actively uses Signals in his or her course, and then intervenes (by flipping the light switch) based on the student's performance data. I specifically asked this question of John Campbell during his ELI 2011 featured presentation in Washington, DC on February 16 (our exchange is captured at time code 00:52:11 of the video archive of his talk).

Essentially, John said he'd made a philosophical decision NOT to present data directly to students because he didn't want professors to be accosted by students with information the prof didn't readily have beforehand.

I really like and respect John a lot, and think I and so many others have directly benefited from the heavy lifting he and Purdue have done in learning analytics before any of us were even looking into this. However, I think his philosophical decision to filter student data through an instructor's lens, and (more importantly) base ALL of the intervention responsibility on instructors, misses the opportunity for self-awareness and self-efficacy you describe for a Corporate Signals.

That's why, when we developed our Check My Activity (CMA) tool, I made the philosophical decision to show students their own data directly, without human intervention, to see if having their data--in the most ideal way for them to understand it--would raise their awareness and motivate them to seek or accept help from instructors or academic advisors. We're just at the start of analyzing how students use the CMA to answer that question. To be sure, some students will under-respond or maybe not at all. But the risk of doing so means you provide a learning environment where some students WILL act on their own. Those are the ones I think we can help, particularly if the CMA can complement the human resources UMBC is all too ready and willing to provide, if only we can connect the students who need help--and want it--with people who can provide it. After all, it's a whole lot easier to answer a request for help than it is to try and instill one in a person, who doesn't know they need it, as a form of Inception. As a result, I like to think of our CMA as a "high tech, high touch" approach to student success and retention.

Apart from tracking and displaying the right metrics that would be meaningful to corporate employees, maybe there wouldn't be so many philosophical concerns, and management can just do what it wants with a Corporate Signals, by edict. But if learning is to be authentic (in any organization), it has to be self-determined somewhat. I applaud your call for an "I/Thou" relationship between the individual and his or her own data. Awareness starts from the inside out, and learning analytics should facilitate and leverage that.


John Fritz