eLearning Roadtrip

Karen Swan and the Amazing Multi-institutional PAR Student Success Matrix

Back in June 2013 the PAR Framework openly published an intervention assessment framework called the Student Success Matrix to help PAR members figure out how to respond when confronted with predictions of which students were likely to drop out. "What should we do to respond to these predictions showing our students at risk?" our members asked.  At that time Dr. Karen Swan, Stukel Distinguished Professor of Educational Leadership at the University of Illinois Springfield, reminded us that there are already lots and lots and lots of things that academic advisors, faculty and other student success professional already do to help students succeed in school. Going back through the professional literature, she and Dr. Peter Shea, SUNY-Albany, helped us find models of student engagement and participation relevant to our work at hand.

Karen proposed a framework to map interventions using predictor behaviors of success and points in the academic completion cycle.  Over the past year Karen and Peter have worked closely with Sandy Daston, PAR Framework Director of Student Success and the entire PAR SSMx working group (Hae Okimoto, University of Hawaii System; Jennifer Freed, Dan Huston, and Fermin Ornelas, Rio Salado College; David Shulman, Broward College; Denise Nadasen, Jack Neill and Karen Vignare, UMUC; Patsy Moskal and DeLayne Priest, University of Central Florida; Michelle Wiley, Penn State World Campus; Mysti Reneau, Community Colleges of Spokane) to refine processes and tools for promoting intervention measurement. This group of experienced educational practitioners and decisionmaker has helped determine what can actually be done for student at risk - even down to knowing how to support those 37 students that are most likely to fail Math 101 this semester. 

PAR institutions have begun the process of inventorying, tagging, describing measuring and evaluating literally hundreds of interventions currenty in use on campuses to help student succeed. Thanks to the SSMx we have accelerated the process of connecting predictions of risk to the interventions that will mitigate risk. 

This post is to acknowledge Karen's leadership and contributions to PAR's thinking about using predictions as cataysts for taking action, and for linking interventions to predictions to underscore the importance of linking predictions to action.

 

When I realized how easy is was to get started with learning analytics, I laughed out loud

I've always wanted to write a headline like that.  

If you are like me, any link suggesting that learning analytics can be easy is worth a quick look. Even if only to sneer at the temerity of whoever would say that. So, thank you for being curious enough to click. Now that I have your attention...

Getting started with a learning analytics initiatve doesn't need to be hard. What's hard is sustaining a learning analytics initiative IF one is only doing something, anything, as a pet project, an exploration, or a trial run because someone gave you free software. 

To paraphrase the incomparable Tina Turner, we're gonna start this learning anaytics conversation nice and easy (don't worry, no dancing required). Here we go. Five quick easy steps to get started:

(1) Start with a real problem to solve. Scope your problem statement so that you maximize your probabiities of  success. Do you have market conditions to address, sneaking suspicions that there might be trouble in River City that you'd better find out about before someone else does? Maybe places where you are pretty sure you can do a better job? You might want to think of something that will help you test the waters of change without putting yourself at risk of total humiliation if it doesn't work.

(2) Figure out who needs to be involved in creating the business case to get this problem solved. How much of a priority is solving the problem you identified going to be for others in your organization? Who need to be engaged and on-board to get this done? 

(3) Determine what you need to answer questions related to solving this problem. What information do you need? Who owns it? Where does it live? How are you going to secure it? How will it be analyzed? Who can see it, work with it, publish it? How will results be used? How long will the results live, and who will have access?

(4) Determine the necessary project talent to make sure you can get the work done. It's not just about having access to people who understand how to work with data. It's about having the right team in place across the board who can execute to plan and deliver results, and making sure that you have planned for enough of their time to be effective. How will you find them? What will you need to attract their attention and keep them engaged?

(5) Develop a bottom-up budget to see if you can afford to do what you want to do. Odds are, you will need more money than you have on hand. (Well... odds are that you will end up needing more money than you think you do, too, but that's a conversation for a later time). So, where might you find the funding to do what you want to do? Who else might be interested in getting involved in your great adventure?

Obviously each one of these steps can take on a life of its own once you get going, but as a place to begin these are a great way to get your arms around your big idea and see if its possible to move ahead.

Now that Predictive Analytics are Hotter than Ever

Now that predictive analytics are hotter than ever in higher education, it's probably a good idea to review a few fundamentals for those who are just starting to sip from the cup of possibilities.

WHY PREDICTIVES? Predictive analytics have been used in a wide variety of settings, including higher education, to manage finances, inventory, operations, assets and resources. Increasingly, higher ed institutions are turning to business intelligence for enrollment management and student recruitment. The next great wave for predictive analytics adoption in higher education is focusing on institutional performance outcomes and individualized student success. 

WHY COLLEGES NEED TO CARE: The more that colleges and universities are being held accountable for achieving specific performance outcomes (e.g., improved student retention, better completion rates for college students), the more predictive analytics will be touted as a solution for anticipating risks likely to be encountered while trying to achieve those outcomes. 

SIMPLY KNOWING WHO IS AT RISK ISN'T ENOUGH. Knowing how to mitigate risks and how different students can be better served with targeted interventions and support makes predictions actionable. Predictions without action don't really matter very much to anyone.

BEWARE THE BRIGHT AND SHINY: It's going to be a little bit like the early days of the first dot.com out there, with lots of  solutions available and even more promises about all the great things that those interventions and solutions can do (!!!). How will you know who to believe? How will you know what's right for you? You owe it to yourself to be informed.

THERE IS NO SINGLE "RIGHT WAY" TO CONDUCT PREDICTIVE ANALYSES. There are many techniques used to conduct predictive analyses. Typically one chooses the technique or techniques likely to yield results for the kinds of predictions one wants to make.

For example, if you are interested in predicting a discrete attribute you might be likely to use techniques including Logistic Regression, Decision Trees (CHAID, CART, Random Forest), Naïve Bayes, Support Vector Machine, Survival Analysis or Neural Networks. If you were more interested in predicting a continuous attribute, you'd look toward techniques such as Multiple Linear Regression, Time series or Decision Trees (CHAID, CART, Random Forest). If you were most interested in common groups, you might consider Hierarchical k-nearest neighbors, Neural Networks or Decision Trees (CHAID, CART, Random Forest).

Selecting the right techniques used for conducting predictive analyses have a lot to do with knowing the questions that the predictions will help answer, or the performance problems that are likely to be solved. Speaking of which...

START WITH A PROBLEM TO SOLVE: Sorry to burst your bubble but predictive analytics don't work quite like Magic 8-Ball. Not even Hadoop technologies do that. Answers simply do not emerge fully formed from the mists of your analytic techniques.  It's easier to focus on solving a problem (e.g. what causes students to drop out? Are these causes common in all settings?) or finding a new opportunity (e.g. what motivates students to complete courses faster?). Your problem statements and queries will help you focus on finding data sources and selecting techniques for analyses that are likely to reveal the patterns you seek.

Back on the Road Trip

Learning anaytics are catalyzing transformative conversations about completion and retention, competencies and ethics, pathways and profiles, micro-credentials and modalities. They are going to have tremendous impact on just about everything that learning technology professionals do. More to the point, they are going to define much of what we do and how we do it. 

You might be wondering what this has to do with elearning. One of the little secrets about predictive analytics is that predictions are useful if one takes actions to address whatever weakeness or opportunities that predictions reveal. Going forward, the real action will not be around predictions, per se, but on taking action to address the risks and opportunities that predictive analytics reveal.

These days this has a lot to do with what "big tent" elearning offers, at least the kind of elearning that delivers on the promise to personlize and motivate and pace, featuring learning technologies from interactive pdfs to wearable apps to immersive environments and adaptive pathways and everything in between.

And with that, we are back on the Road Trip.

Living Under the Sword of Data, on #Lrnchat

On May 9, 2013 I will have the pleasure of joining the crew at #Lrnchat for a session on the topic of learning analytics. You will find us on Twitter, Thursday, 8:30 pm ET/5:30 PT. Look for us at #lrnchat, follow along at @lrnchat.

 #Lrnchat is a regularly scheduled event that takes place every Thursday evening.  Participants using any number of tweeting tools (such as Tweetdeck and Tweedgrid) to keep track of the messages that fly around among the session participants. It is hosted and facilitated by a number of learning and development social media experts who shall remain nameless until I get their permission to drop their names. Questions get asked, responses get posted, conversations get started, friendships get launched. It's amazing how much one can pack into 140 characters! 

The name of this particular post, in anticipation of tomorrow's #Lrnchat, comes from one of my favorite analytics parables. In March of 2009, the lead designer from Google resigned in protest to what he described as an engineering culture that was obsessed by numbers. Douglas Bowman offered his reasons for resigning  in a blogpost entitled "Goodbye Google". In his explanation he gave the following examples of how data-obsessed Google had become:

"Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such minuscule design decisions. There are more exciting design problems in this world to tackle."

In closing, he wrote that "I’ll miss working with the incredibly smart and talented people I got to know there. But I won’t miss a design philosophy that lives or dies strictly by the sword of data."

I remember being struck by that phrase as the perfect way to describe the double-edged benefits and risks that analytics bring to discussions of learning, achievement, progress, success. I offer this as something that each of us should keep in mind as we gravitate toward a greater use of enterprise data in our decision-making. 

I know we will have 90 minutes to talk about this. In anticpation of the conversation can I just offer this observation - It helps to have an idea about the problem you are trying to solve before you dive into an analytics campaign. And once you DO know, you need to be prepared to respond. Problem identification, without taking action, is almost worse than not knowing at all.

Here are a few references that I hope you will find to be useful. Happy reading! See you on Twitter.

Background References

Goodbye, Google The blogpost written by designer Douglas Bowman when he resigned from Google in protest to the data culture around decisionmaking. http://stopdesign.com/archive/2009/03/20/goodbye-google.html

The Gawker story that picked up the story of event leading to Bowman’s resignation from Google http://gawker.com/5177144/googles-data-fetish-drives-away-its-top-designer

Articles I've written on learning analytics:

http://www.educause.edu/ero/article/data-changes-everything-delivering-promise-learning-analytics-higher-education

http://www.astd.org/Publications/Magazines/TD/TD-Archive/2012/08/8-Realities-Learning-Professionals-Need-to-Know-About-Analytics

Recent tweets on Big Data –related topics by @edwsonoma

The Big Data Landscape http://bit.ly/12KXpvA 

11 Big Data myths http://bit.ly/15CD26B 

The Surprising Predictive Power of Analytics http://lnkd.in/M2VS9j 

The Rising Costs Of Misunderstanding Big Data http://lnkd.in/fBGWVP 

Ten things you need to know about Big Data http://tinyurl.com/cabxeam 

What's the skillset of a talented #BigData Scientist? http://ow.ly/kNKw7 

 

The Big Data Landscape

If you post the term "Big Data" in your Google search window today, more than 2 billion google search results will appear. If you perform a similar search on Bing, you will see that more than 1,790,000 Big Data links have been added to the Internet in the past 24 hours alone. Clearly, there is big interest in big data.

This graphic by Matt Turck and Shivon Zilis is an exceedingly helpful image for understanding the breadth and complexity of the Big Data landscape. Matt Turck's blogpost  also provides great information for understanding some of where this emerging market is taking us. 

 

 

For those of us working in post-secondary education circles, the emergent interests in big data, analytics, learning analytics and related evidence-based decision-making endeavors have been inspired by possibilities of better experiences, more personalization and better behavioral targeting. Still most education stakeholders who are giving learning analytics some consideration may be surprised to see the depth, breath and sophistication of the big data world.  While WCET's work on the PAR Framework has certainly contributed to a growing awareness on the power of predictive analytics for academic risk identification and the value of collaborative, multi-institutional data mining to improve student success, we are mindful that our student success focus touches a relatively narrow, targeted part of the Big Data picture. This graphic offers a more complete picture of the total ecosystem.

NEW NSF REPORT: New Technology-based Models for Postsecondary Learning

In January 2013 I  participated in a workshop sponsored by the National Science Foundation to help establish a 10 year vision to reconceptualize postsecondary learning. This past week our co-convenors, Chris Dede, Harvard University and Eric Grimson, MIT, released the 47 page report that captured the results of that meeting. Entitled New Technology-based Models for Postsecondary Learning: Conceptual Frameworks and Research Agendas, this is a comprehensive overview of promises, opportunities and challenges that will keep all of we post-secondary learning stakeholders on our toes for the next decade.

You can download the working draft of this paper here.

The basic premises addressed include the following:

• Advances in technology and in knowledge about expertise, learning, and assessment have the potential to reshape the many forms of education and training past matriculation from high school.
• In the next decade, higher education, military and workplace training, and professional development must all transform to exploit the opportunities of a new era, leveraging emerging technology-based models that can make learning more efficient and possibly improve student support, all at lower cost for a broader range of learners. 
• Potential risks must be managed, including those arising from the disruption of established delivery economics in our current learning institutions, the variable quality of learning outcomes these new models offer today, and the technical and conceptual challenges of better understanding how to design, develop, and implement innovative capabilities in ways that reliably deliver on their promise.

Workshop participants developed a framework for understanding this sea change and sketched steps towards a research agenda for realizing its benefits while avoiding pitfalls.

Contributors to this report include Daniel E. Atkins, W.K. Kellogg Professor of Community Information, and Professor of Information and Computer Science, University of Michigan - Ann Arbor. Lori Breslow, Director, Teaching and Learning Laboratory, MIT. John Cherniavsky, Senior Advisor, Division of Research on Learning, NSF. J.D. Fletcher, Senior Research Staff Member, Institute for Defense Analyses. Diana Oblinger, President and CEO, EDUCAUSE. Roy Pea, David Jacks Professor of Education and Learning Sciences, Stanford University. James W. Pellegrino, Distinguished Professor and Co-Director, Learning Sciences Research Institute, University of Illinois at Chicago. Bror Saxberg, Chief Learning Officer, Kaplan, Inc. James H. Shelton III, Assistant Deputy Secretary for Innovation and Improvement,U.S. Department of Education. Russell Shilling, CAPT, MSC, USN, Program Manager, DARPA. Greg Tobin, President, English, Math, and Student Success, Pearson Higher Education and yours truly, Ellen Wagner, Executive Director, Western Interstate Commission for Higher Education Cooperative for Educational Technologies. Arthur Josephson, Harvard Graduate School of Education, was our researcher.

The Reason THIS EDUdotcom May Not Be As Different From Last Time As You Think

If there was ever any doubt that we are in the midst of a new dot.com boom for education, that doubt was removed at this week's SXSWEDU. The event brought entrepreneurs and educators to Austin, Texas for four days of panels and a competition for education start-ups. I had the great fun of participating in a panel discussion on "Are Courses a Commodity?" with Myk Garn, Mickey Revenaugh and Michael Horn. Vanessa Dennen and I helped Curt Bonk rehearse his "cage match" answers while sharing beers and ribs at one of the Cengage social events. I enjoyed hearing Bill Gates speak to the crowd about his vision for a transformed education system. It was definitely an interesting mix of energies.

Gotta say, not all of it felt all that good. This morning's Chronicle of Higher Education led with a headline that explicitly named the tensions between entrepreneurs and educators. You can read the Chronicle report here.

While I was there, I kept hearing that THIS dot.com is different because it's about education, and because the tech is better and data will inform us and investors are smarter and the market is ready, and there are business plans and people aren't just throwing exuberant ideas at the wall and hoping that something sticks. 

Really? 

I have to say that the confident assurances that this educational technology boom is different just didn't make me feel any less skeptical. Because in the same way that there are some who really DO believes that MOOCs are the birth of online learning, it is clear that there is an entire generation of investors and entrepreneurs who really DO seem to believe that the ideas thrown against the wall of the 1990s are somehow less legitimate than some of the ideas that are currently being thrown against the wall today.

So let me tell you why this particular new EDU.dot.com isn't as different as visonaries and investors would have you believe.

Ready?

It's because no matter how new your tech is or how great the idea is, or how impressive the possibilities are, or the circumstances or the bandwidth or the platform or the operating system we won't figure out how to crack the code on transformation until we change the most important part of the equation. And that is the human factor. Eventually it comes down to people being ready to embrace the change. The ability to ride out the hype cycle and get oneself to a true plateau of productivty will all come down to the degree we can induce people to change their ways, completely rethink their practices, and help them figure out how to use new tools toys, apps and the like deliver on their promises.

The excitement around innovative technology futures for teaching and learning has energized educational researchers to think broadly and deeply about the possibilities they represent.  The venture capital communities’ recognition that education may be ready for its “Internet moment” has also generated massive interest in developing promising ideas for products with commercial consideration.

Between these two exciting arenas of exploration live equally important opportunities for transformation that come from practice-focused solutions contributing to demonstrable improvements in student engagement, faculty performance, and institutional accountability. 

People. We've got to get ready, too.

---

[1] 

 

 

The Burden of Knowing

Important questions can get swept aside when the rumble of excitement about future possibilities are loudest. Nowhere is this more clear than in the emerging arena of learning analytics.

The fact that we can track information just about anytime, anyplace and on any device boggles the mind with possibilities. But it also begs a few questions: Does the fact that we will eventually be able to collect, track, link, visualize, tokenize, correlate, factor or parse data pulled through APIs mean that we should? When we can collect everything, will we know what to look for when applying various analysis technique among the research protocols? Will we recognize the answers we are looking for when we see them forming in the mists or hanging from the branch of a CHAID tree?

Perhaps even more to the point - will we be able to do anything meaningful with what we find? What are we going to DO, once we know?

The notion that all learning should be tracked, or that there is value to tracking activity stream level information, or that Hadoop-like pattern-seeking technologies will serve the needs of the learning enterprise in meaningful ways, must also consider what that capability is really worth to the enterprise. 

It  presumes that once we DO recognize new patterns of loss, momentum and opportunity that we are going to be able to do something about what has been found. 

But most people do not understand how long it is going to take the learning world at large to think of data as his or her friend. To see the results of analyses as benchmarks for better understanding performance, not as red flags requiring an intervention. One can only wonder how many enterprises will be slow to collect learning data worrying that they can't afford the Burden of Knowing what their data will reveal. 

 

 

 

Funsuckers

Marc Prensky once famously accused instructional designers of sucking the fun out of learning at an EDUCAUSE Learning Initiative Conference a number of years back. As you can imagine, instructional designers were not amused. 

I was reminded of Marc's quotable quote during a recent conversation regarding the value and role of structure and specifications - or the lack thereof - for designing learning experiences. This is a more nuanced conversation than differentiating between formal and informal learning, or whether one is online or on-the-ground. This also includes things that lead to immersion, augmentation, mobility, and personalization. It inevitably leads to thoughts of learning analytics. Life-long learning tracking even.

It's almost as if today's emergent designers want to dance on ADDIE's grave and declare that ID funsucking is a thing of the past. Which brings me to my point.

It is inspiring to know that there are exceedingly passionate designers out there who want to create learning experiences that engage and inspire, helping the world see new possibilities with fresh eyes. Good on ya. The world needs you. 

But designers also must bring order to chaos. Most of us work in settings where we are mostly responsible for the practical, pragmatic applications of design principles to achieve tangible measurable results for individuals AND enterprises.  It does not matter if we believe something to be true with total heart and soul if it cannot demonstrate tangible value of those beliefs to stakeholders. Not in the future. Now.

So here is a message for the "true believers". When someone tells you than an idea is a little early for their stakeholders....it doesn't mean he or she "don't get it", nor does it necesarily mean that she or he is a pin-headed, short-sighted idiot. It simply means that the case still needs to be made.

To be clear, if it weren't for people believing that they have a better way of doing something and being willing to social the ideas ad nauseum we probably *would* still be sniffing ditto copies in our obligatory F2F training sessions.

All *I'm* saying is that visionary awesomeness on its own probably isn't enough. Sometimes it might just take a few funsuckers to spin all that straw into gold.