eLearning Roadtrip

Living Under the Sword of Data, on #Lrnchat

On May 9, 2013 I will have the pleasure of joining the crew at #Lrnchat for a session on the topic of learning analytics. You will find us on Twitter, Thursday, 8:30 pm ET/5:30 PT. Look for us at #lrnchat, follow along at @lrnchat.

 #Lrnchat is a regularly scheduled event that takes place every Thursday evening.  Participants using any number of tweeting tools (such as Tweetdeck and Tweedgrid) to keep track of the messages that fly around among the session participants. It is hosted and facilitated by a number of learning and development social media experts who shall remain nameless until I get their permission to drop their names. Questions get asked, responses get posted, conversations get started, friendships get launched. It's amazing how much one can pack into 140 characters! 

The name of this particular post, in anticipation of tomorrow's #Lrnchat, comes from one of my favorite analytics parables. In March of 2009, the lead designer from Google resigned in protest to what he described as an engineering culture that was obsessed by numbers. Douglas Bowman offered his reasons for resigning  in a blogpost entitled "Goodbye Google". In his explanation he gave the following examples of how data-obsessed Google had become:

"Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such minuscule design decisions. There are more exciting design problems in this world to tackle."

In closing, he wrote that "I’ll miss working with the incredibly smart and talented people I got to know there. But I won’t miss a design philosophy that lives or dies strictly by the sword of data."

I remember being struck by that phrase as the perfect way to describe the double-edged benefits and risks that analytics bring to discussions of learning, achievement, progress, success. I offer this as something that each of us should keep in mind as we gravitate toward a greater use of enterprise data in our decision-making. 

I know we will have 90 minutes to talk about this. In anticpation of the conversation can I just offer this observation - It helps to have an idea about the problem you are trying to solve before you dive into an analytics campaign. And once you DO know, you need to be prepared to respond. Problem identification, without taking action, is almost worse than not knowing at all.

Here are a few references that I hope you will find to be useful. Happy reading! See you on Twitter.

Background References

Goodbye, Google The blogpost written by designer Douglas Bowman when he resigned from Google in protest to the data culture around decisionmaking. http://stopdesign.com/archive/2009/03/20/goodbye-google.html

The Gawker story that picked up the story of event leading to Bowman’s resignation from Google http://gawker.com/5177144/googles-data-fetish-drives-away-its-top-designer

Articles I've written on learning analytics:

http://www.educause.edu/ero/article/data-changes-everything-delivering-promise-learning-analytics-higher-education

http://www.astd.org/Publications/Magazines/TD/TD-Archive/2012/08/8-Realities-Learning-Professionals-Need-to-Know-About-Analytics

Recent tweets on Big Data –related topics by @edwsonoma

The Big Data Landscape http://bit.ly/12KXpvA 

11 Big Data myths http://bit.ly/15CD26B 

The Surprising Predictive Power of Analytics http://lnkd.in/M2VS9j 

The Rising Costs Of Misunderstanding Big Data http://lnkd.in/fBGWVP 

Ten things you need to know about Big Data http://tinyurl.com/cabxeam 

What's the skillset of a talented #BigData Scientist? http://ow.ly/kNKw7 

 

The Burden of Knowing

Important questions can get swept aside when the rumble of excitement about future possibilities are loudest. Nowhere is this more clear than in the emerging arena of learning analytics.

The fact that we can track information just about anytime, anyplace and on any device boggles the mind with possibilities. But it also begs a few questions: Does the fact that we will eventually be able to collect, track, link, visualize, tokenize, correlate, factor or parse data pulled through APIs mean that we should? When we can collect everything, will we know what to look for when applying various analysis technique among the research protocols? Will we recognize the answers we are looking for when we see them forming in the mists or hanging from the branch of a CHAID tree?

Perhaps even more to the point - will we be able to do anything meaningful with what we find? What are we going to DO, once we know?

The notion that all learning should be tracked, or that there is value to tracking activity stream level information, or that Hadoop-like pattern-seeking technologies will serve the needs of the learning enterprise in meaningful ways, must also consider what that capability is really worth to the enterprise. 

It  presumes that once we DO recognize new patterns of loss, momentum and opportunity that we are going to be able to do something about what has been found. 

But most people do not understand how long it is going to take the learning world at large to think of data as his or her friend. To see the results of analyses as benchmarks for better understanding performance, not as red flags requiring an intervention. One can only wonder how many enterprises will be slow to collect learning data worrying that they can't afford the Burden of Knowing what their data will reveal. 

 

 

 

Losing Our (Analytics) Religion

You may have noticed that I've been kinda quiet on the blogging front. It's not for lack of things to say, exactly. More that there has been so much sound and fury around the emerging topic of learning analytics that I've wanted to listen to what was going on before I added my .02.

But there have been a few things on my mind, especially now that we are well into the second year of our work on the PAR Framework. And we are coming up on fall conference season, where I expect we will all be hearing a lot about learning analytics solutions.

As you gird yourself for the coming onslaught of emerging thought leadership that will swirl around us regarding learning analytics, big data, and the like, just remember: Yes, it's hard to turn mashes of clicks into anything meaningful when you are dealing with something like learning, which, like it or not, is like a religion to many people. Anybody who tells you otherwise does not understand your work. 

Don't be shy about pushing back a little bit and asking for clarifications. The answers to learning questions - enterprise, individual or otherwise - are not likely to emerge fully formed from the mists of Apache Hadoop, Google Analytics or Adobe Omniture. All of which are very cool technologies. But seriously, they can't make up your mind for you.

It's not the data, the metadata or the paradata, nor is it necessarily the analytics we perform, that matters the most. What matters is what we DO with the information coming from these analyses, that will help us achieve results that matter to our stakeholders.

Since You've Asked: More About the PAR Framework

The December 15, 2011 Chronicle of Higher Education's Wired Campus blog by Marc Parry shared news about a groundbreaking project we have taken on at WCET.  It describes our efforts to create a single, federated database of de-identified student records contributed by multiple universities and colleges. Six forward-thinking WCET member institutions have worked together this past six months to see if it was possible to analyze variables common across these institutions to look for patterns that can inform us about student loss and momentum. I'm pleased to say that we have succeeded. Baby steps. But still.

We've received a lot of questions. Here are a few answers. There will be more.

This is a multi-phased project. We are concluding our work on Phase 1. The best is yet to be.

Our first phase focused on demonstrating that it is posssible to get multiple postsecondary institutions here in the US to contribute their data to create a single unified data set. We are now applying a variety of descriptive, inferential and predictive analyses to look for patterns showing variables likely to affect student success. 

Phase 1 = Proof of Concept. Creating a single federated data set from the de-identified student records from six unique institutions has never been done before. 

We've started with 6 schools. We are adding additional schools as we continue our efforts to build out the federated data set. While 640,000 students records and 3.1 million course records are a lot more than most of us are used to working with, it is still just a sample from 6 schools. 

We've started with 34 variables that are common across each of the institutional partners. Much of our current work has focused on establishing common definitions of these variables to ensure that comparisons and aggregations are valid, reliable and repeatable. There will be more variables.

We are currently focused on online programs. Just about every single transaction associated with an online program is digitally captured and stored somewhere. It seemed the obvious, self-evident place to begin.

We've focused on pattern strategy exploration, not null hypothesis testing (at least not in the conventional sense). 

We're really not trying to answer research questions using big data analysis techniques. Not yet. We'll get there. First things first. The journey of a thousand miles begings with the first steps.

More to the point...if we do our work right, then perhaps some day YOU will be answering YOUR research questions using the federated data set using big data analysis techniques.

The team working on this effort brings expertise from institutional research, statistical analysis, quant and qual research, cognition and instruction, online learning, higher ed administration, educational public policy, project management, and commercial software development. They come from the public and for-profit sectors. Then are from community colleges and 4 year comprehensives. Several of our institutional partners are already award-winning, acknowledged practice leaders in the hot, hot, hot domain of learning analytics. These are some of the smartest, most dedicated, creative, committed people I know. 

None of us - not WCET, not the core team of experts, not any of our fabulous institutional partners - are making any kinds of reccommendations from our findings yet. You may have noticed that my quote in the Chronicle was about as non-specific as one can possibly be. But be assured, there are going to be some very interesting implications for practice and policy that will emerge as the new patterns we are finding are more sharply focused in the coming weeks and months. 

Yes of course we are going to share our findings. Sharing is a core WCET value. We exist to share. That's what Cooperatives do.

 

20 "eLearnings" From Two Months on the Road(trip)

Been a busy couple of months of elearning road-trips. From EDUCAUSE to Online EDUCA, from WCET to WICHE, from DevLearn to AECT (with a little bit of virtual Sloan-C tossed in for good measure). Literally thousands of plenaries, presentations, and poster sessions from which to choose, along with hundreds of vendor stands and trade show booths. Here are some of my big take-aways:

• Hooray and congratulations! This distance/online/elearning thing we do has tipped into mainstream consciousness. Let's give a cheer for the researchers, the teachers, the authors, the artists, the technologists, the producers, the analysts and the evaluators who have made this happen. We rock.
• We matter enough that governments want to regulate us. Congressmembers rail against us. No more flying under the radar.
• Analytics are hot, hot, hot. Learning analytics, content analytics, web analytics, engagement analytics, to name a few.
• Speaking of analytics...mathematically speaking, a learning outcome is pretty much the same as a point of sale. (Thank you, Phil Ice.)
• Second Life is dead, dead, dead. Sorry.
• Mobile is the future.  Smartphones and tablets and readers, oh, my.
• Game based learning has a long way to go before people take it seriously.
• High quality video conferencing is making a comeback.
• We may get inspired by big ideas, but we are moved to action by fear, uncertainty and dread.
• On the commercial front...Pearson intends to own our universe. Adobe has pretty much given up on us. Just about everybody wants to sell us an "analytics solution". Or a mobile app.
• People really need to quit thinking that "open" is the same as "free".
• It's a lot easier to pontificate about the importance of change than it is to do things differently.
• There is a lot of research out there. Some of it is useful. 
• Empirical evidence doesn't need to be quantitative...but it does need to be a little more robust than simply pointing to people who are doing the same things.
• Quality is rarely absolute. It's contextual and situational. It may make more sense to think about "effective practices" rather than "best practice".
• Practitioners who make big sweeping statements without doing any background work on what others may have already been doing, saying or thinking end up sounding just a little bit naive and foolish. C'mon, people...use The Google.
• On the Internet, nobody knows you're a dog...but EVERYONE knows if you are not prepared.
• One can pack a lot of snark in 140 characters.
• Learning is as natural as breathing. Learning really isn't the problem. Enabling meaningful, intentional outcomes and creating experiences that engage and inspire without costing an arm and a leg and without killing our interest and enthusiam for learning is the problem. 
• Sometimes you just need to show up.

Lessons Revisited: RADAR, Pearl Harbor and Paying Attention to Evidence

One of our Sage Road Solutions LLC clients, the Western Interstate Commission for Higher Education, is holding its biannual meeting at the BYU Hawaii campus on the island of Oahu this week. Having spent the past several days in Denver hosting the annual meeting of the WICHE Cooperative for Educational Telecommunications, known as WCET), I'm still running on Mountain time. I woke up early, so I decided to take advantage of being in this glorious location to take a walk along the beach.

About 200 yards from the hotel an historical marker notes the location of the Opana mobile RADAR site. The story accompanying the marker offers an interesting object lesson for those of us trying to bring innovative applications of technology to the learning technology metaverse.

According to Wikipedia the term RADAR was coined in 1940 by the U.S. Navy standing for radio detection and ranging. It was brand new technology, showing enough promise that pilot programs were launched to test its utility. In 1941, a number of mobile radar trucks were deployed on Oahu to keep an eye on the skies near Hickam Field.

 

Early on the morning of December 7, 1941, two soldiers stationed at the mobile radar site near Opana Point saw electronic blips of what looked like many planes approaching the island. Reporting their findings, they were told that what they were seeing were *probably* planes expected from California, and not to worry about it.  Consequently this information was never reported to those in command. Shortly thereafter, the Japanese planes detected by the RADAR bombed Hickam Field and Pearl Harbor, destroying US ships and planes, leading to the US's declaration of war on Japan and igniting US involvement in World War II.

The historical information was certainly interesting in its own right. The narrative that accompanied the marker, however, was particularly striking:

"The failure to warn the Army or Navy command in Hawaii was not a failure of the technology as much as it was a failure of organization. While the technology of RADAR functioned as intended and detected the incoming planes, there was no way to accurately assess the information and communicate this knowledge to those in command." 

So, let's fast forward to October, 2011. On Thursday of this past week the WCET team working on our Predictive Analytics Reporting (PAR) Framework project revealed some of the first findings from our ground-breaking big data project. At the risk of sounding hyperbolic, the descriptive, inferential and predictive techniques being applied to this massive data set are providing some eye-opening perspectives on variables affecting student loss and momentum. As noted in a Forbes.com online article featuring Phil Ice, our project's principal investigator (who, by the way, really hates being called "the other Dr. Phil"...so please don't) these results are going to rock the way that educational decision-makers think about everything from college admissions, student services, technology investments and even "best practices" for online learning. 

Still, I have a feeling that these "big data" results will go through a period of being pooh-poohed by decision-makers who will have a hard time letting go of null hypothesis research methods. Traditional researchers' disdain for the so-called data mining techniques associated with pattern strategy and business intelligence used in other sectors are likely to get in the way. For many traditional researchers, educational data mining is a little bit like a stupid pet trick - even though these same folks really seem to like the convenience of Amazon shopping recomendations, Netflix movie recommendations and Pandora music recommendations.

I can't help but wonder if the findings about student loss and momentum that the cracker-jack research crew are sharing will be disregarded very much in the same way that Army and Navy command dismissed the evidence that Pearl Harbor was in harm's way because the techniques are too new to be taken seriously. I just hope that we can figure out how to help people pay attention to evidence before it is too late.

 

Big Data Comes to US Higher Education

WCET, the WICHE Cooperative for Educational Technologies, today announced the successful federation of datasets from the six institutions participating in the PAR (Predictive Analytics Reporting) Framework proof of concept project, just four months after the project’s early June launch. The goal of the PAR Framework is to identify variables that influence student retention and progression and to determine the impact of various demographic data on factors influencing loss and momentum. The data will be used to explore patterns that emerge when the datasets from considerably different institutions are analyzed as a single, unified sample. 

The dataset includes over 640,000 anonymized student records and over 3 million course level records, focusing on 33 common variables. The institutions represent public/private, two-year/four-year, and publicly-funded and proprietary institutions. 

“We have been using analytics as a tool for improving student retention and progression for several years,” said Dr. Phil Ice, vice president, research & development, American Public University System, and principal investigator of the PAR Framework Project. “Unifying records from multiple schools has the potential to expose generalizable patterns that can provide guidance to institutions across the board, regardless of their internal level of analytics expertise.” Ice continued, “In higher education, no predictive analytics project has been able to use a multi-institutional sample of this size. As a researcher, having a sample size of this number changes everything.” 

The WCET member institutions engaged in the PAR Framework project include American Public University System, Colorado Community College System, Rio Salado College, University of Hawaii System, University of Illinois Springfield, and the University of Phoenix. Each institution obtained IRB (Institutional Review Board) approval. All data was anonymized and encrypted to ensure that no data are personally identifiable. 

“We had been warned that we could never find six schools willing to collaborate on something as sensitive as an institutional student and course level data federation project,” noted Ellen Wagner, WCET Executive Director. “It’s exciting to see how forward-thinking WCET member institutions are bringing the power of “big data” to post-secondary education in the US. The magnitude of the data set unprecedented and the diversity of the participating institutions will provide us granular insight into similarities and differences across higher education institutions in the for profit, public and community college segments.” 

A summary of first findings are scheduled for presentations at the 23rd Annual WCET Conference in Denver, Colorado October 27th (https://wcetconference.wiche.edu/).

Thought You Might be Interested...

...in some of the thoughts I shared with my WCET crew today about learning analytics and lessons we can learn from Moneyball. It's an extension of my eLearning Roadtrip post on the same general topic.

I suppose this gives you an idea of what is top-of-mind for me these days. Well, that and wanting to raise the bar on online learning excellence. Simple pleasures.

If you are interested in taking a look at my WCET post, please click here. 

While there, please take a look around at what the fine folks at the WICHE Cooperative for Educational Technologies are doing these days. If you happen to care about accelerating the adoption of learning technologies innovation in practice you might find that WCET is engaged in work that matters to you.

As a matter of fact....WCET's Annual Meeting is being held in Denver October 26-29. We will be sharing some of the preliminary results from our Predictive Analytics Reporting Framework, along with lots of other information about learning analytics and other emerging topics of interest to online learning and learning technology professionals. Might be a great time to come see what the WCET buzz is all about.

What Learning Professionals Can Learn from "Moneyball"

Today is the premiere of "Moneyball", the long anticipated movie starring Brad Pitt. Based on the book by the same name, this is a story of how the Oakland Athletics baseball team actively leveraged player statistics as business intelligence, and how this allowed them to successfully compete against the huge payroll differentials among just about every other Major League Baseball team. In spite of the fact that they had significantly less money to pay the players they hoped to recruit, the Oakland A's developed the art of winning into a science, and changed baseball in the process.

Bucking the long accepted, "know 'em when you see 'em" approach to player recruitment used pretty much by everyone in baseball, Manager Billy Beane and his team of MBAs figured out how to use broadly available player statistics to find and recruit the highest performing prospects as reported by their statistics, not by the usual folk wisdom favored by scouts. In many cases, the statistical high performers were complete unknowns, not necessarily looking anything like the standard kind of player scouts tended to favor. By virtue of their unknown status, the A's were able to sign them at salaries the team could actually afford to pay. They were able to keep them long enough to foster talents. And while many an Oakland A team member would go on to great big huge contracts with other teams, they did so after mastering ther craft while wearing an A's uniform.

I was so captivated by the analytics possibilities from the movie buzz (yes, I know. I'm a nerd) that I went out and bought a copy of the book Moneyball by Michael Lewis at my local bookstore over the weekend. It just seemed like there are things that education types could learn about using stats as part of a decision strategy. It was a fast compelling read. I have to tell you, there are many things that those of us starting to think about using analytics to improve our practice can learn from this story.

Without spending a lot of time on the details I thought this one quote might resonate a bit with those among us who are up to our eyeballs in making the case for analytics:

"Baseball keeps copious records, and people talk about them and argue about them and think about them a great deal. Why doesn't anybody use them? Why doesn't anybody say, in the face of this contention or that, "Prove it'?"

Which of course is exactly what the A's went off and did.

Now. What if we substituted the words "postsecondary education" for "baseball"?

In education we would need be content with demonstrating tenability at a particular level of significance...but still.

There is a lot to learn here. Things like, letting evidence speak more loudly than opinions. Giving less well funded, smaller competitors a shot at a level playing field by using information as intelligence. Using the information we have at hand to help us compete and succeed.

What? Let the evidence speak for itself? Madness.

Of course, the prospect that Brad Pitt is going to (1) make statistics sexy by (2) showing a real-world example of (3) using data-driven decision-making to (4) completely transform a well respected American institution is making this little ol' quant research analyst do the happy dance.

When I Think of the Future of Online Learning

I am attending a Summit later this week. The University Professional and Continuing Educational Association and the Association for Continuing Higher Education are co-hosting a sold out invitational meeting to consider the Future of Online Learning. Many dozens of learning leaders will be in attendence. I am looking forward to hearing what everyone has to say.

As a long-time online learning / elearning / distance learning / teletraining kinda gal, these kinds of events please me in an odd, parental sort of way. It is quite satisfying to see how over the past more than 25 years of grinding their way into teaching practice, learning technologies now routinely extend the reach of the traditional classroom. Online learning has tipped into mainstream consciousness, even if mainstream exceptance continues to be somewhat elusive. 

Since I DO have some skin in this online learning game, let me tell you some of the things I hope we will hear about during the next few days:

• I hope we will hear that it is time to move past online learning as a technological analog of the traditional college classroom.
• I want us to talk openly about getting serious about high quality, online learning experiences that scale.
• I want to hear how more students will have the opportunity to achieve the dream of a college education.
• I want to hear more about the $10,000 degree. Or the $5,000 degree. Either one.
• I want to hear how we will make sure that online students can quit being ashamed to say they got their degree from an online school.
• I want to revisit how adult learners demonstrate that the knowledge they have accrued outside the classroom is a legitimate part of competency certification.  
• I want to hear how the campus learning portal / LMS is being deployed in new and innovative ways, 
• I want to hear how online learning professionals are going to use descriptive, inferential and predictive analysis of de-identified student records to better predict when students might benefit from a bit of extra help to stay on academic track.
• I want to hear how institutions are going to make online learning experiences more compelling,  engaging, personalized and professionally relevant.
• I want to hear that there is room for informal collaborative learning moments in every student's program of study.
• I want to see more attention paid to interactive digital learning content - PDFs and digital textbooks, apps and learning objects - that students produce and publish.
• I want to understand how OER will sustain itself.
• I want to understand how mobility is changing the online learning game, and how social media bring real-time interpersonal interactivity to our conversations.
• I want to see the use of game mechanics and gamification extend our thinking about how to motivate and engage learners without turning games into the next generation of crappy elearning.

From everything I can see on that horizon these issues are all going to be part of the future of online learning. With respect, they are already part of the present. IMHO, it's time to make dealing with these issues part of our practice. Looking forward to what comes next.