eLearning Roadtrip

Driving the #LearnerAnalytics Bus

The message emerging from this week's University of Maryland University College (UMUC) Learner Analytics Summit (#innovatelearning) was explicit and to the point: Forward thinking instititutions are stepping up to opportunities for improving post-secondary experiences for students using data to inform proactive, evidence-based decision-making. 

No doubt, we all hope that the answers to questions we can't sometimes even articulate yet are going to emerge from dashboards and apps, fully formed, as if from the Magic 8-Ball. Sadly this is not the way things work. We need to build capacity for decision-making in a data-informed world just as much as we need to figure out the exo-structure required for systems to interoperate and for intelligence to be shared.

Matt Pistilli (Purdue University) tweeted one of the great tweets of the event when he noted that "the data do not speak for themselves. I have, in fact, been in the room with them, and they did not speak a word #innovatelearning".  It's a great reminder that if we are going to get anywhere quickly on this journey, we are going to need to be the ones driving the bus on learner anaytics adoption and creating organizational cultures of evidence-based decision-making.

Malcolm Gladwell has talked about tipping points, those moments when a new practice or innovation, idea or trend emerges into public consciousness. Learner analytics are at the point of tipping into public consiciousness in the war on student loss.  

Congratulations to Karen Vignare and her team for convening us, to Bill Moses and the Kresge Foundation for the means to do so, a high-five to Linda Baer for her guiding hand, and thanks to all the presenters and participants for helping us start figuring out how to drive the learner analytics bus, on the journey toward Evidence-Based Decision Land. (Cue unicorns, bluebirds and rainbows.)

PAR Framework: Learning Analytics as a Service for Higher Ed

Thanks to Libby Nelson's great article Big Data 101: Colleges are hoping predictive analytics can fix their dismal graduation rates, we have been fielding a lot of questions about the PAR Framework. Here is an update on who we are and the work in which we are engaged.

MEET THE PAR FRAMEWORK - The Predictive Analytics Reporting (PAR) Framework is a national, non-profit provider of learning analytics as a service. PAR brings 2-year, 4-year, public, proprietary, traditional and progressive colleges and universities together to collaborate on mitigating student loss by identifying the effective practices that support student progress toward their academic goals. The PAR Framework offers a unique multi-institutional perspective for examining dimensions of student success that will help improve retention in US higher education.  PAR is distinguished among the many data analytics solutions emerging in the education domain by its common, openly published data definitions and student success frameworks.

HOW PAR WORKS - PAR uses predictive analytics to deliver tools and reports aimed at improving academic success for students in online, blended and on-ground programs. PAR members receive flexible predictive models, cross-institutional benchmarks, and an intervention insight toolkit supporting a student success methodology that links predictions with interventions and student supports.

• PAR institutions provide a full set of anonymized undergraduate student data that allows for comparative investigation of student success trends over time, at the individual student, course and degree level. (PAR members provide periodic updates of their data ensuring the ability to measure changes over time, the impact of student interventions and enabling the predictive models that PAR produces to be adjusted and tuned for current data.)

• In return, PAR institutions receive actionable reports and student watch lists developed using predictive analytics.  Once institutions deliver their student-anonymized and institutionally de-identified data, PAR researchers and data scientists use statistical and data mining techniques to identify and reveal the factors that most impact student success in the combined dataset and the individual institution level.  Members quickly receive institutionally-specific models that predict the likelihood of each student achieving institutional specific milestones. These include but are not limited to passing a course or courses and being retained at the institution.  Member institutions also receive academic models for their institution that include detailed student level data to create watch lists, structure interventions and inform local decisions about the factors identified as having the greatest impact on student success. They also receive institutionally anonymized and de-identified findings from the entire dataset that yield over-arching insights about retention patterns in U.S. higher education to inform practice and policy.

PAR TOOLS FOR BENCHMARKING:

• The PAR Student Success Matrix (SSM^X) helps institutions comprehensively assess their student success policies, interventions and programs. The SSM^X makes it possible to organize the wide variety of student supports – from orientation to mentoring to advising – into a systematic validated framework designed to quantify the impact of student success practices and determine the best support for students at the point of need.  The SSM^x also reveals gaps and overlaps in student support programs and gives institutions the tools to evaluate the efficacy of their investments at the program level.  
•  Quantified intervention effectiveness results. The common PAR measures for assessing and predicting risk and the validated frameworks categorizing student support services create the mechanism to effectively measure the impact of student supports within and across institutions.  PAR member institutions are encouraged to take part in quantitative, closed loop field tests, measuring the effect of targeted student supports (e.g. tutoring, student services, email, text messages alerts) against at-risk students.
• Institutional benchmarks. PAR Framework members receive institutionally-reflective and cross-institutional benchmark reports of student level outcomes such as retention, successful credit accumulation, academic performance indicators, and earning a credential.

COMPREHENSIVE INSIGHTS BASED ON VARIATIONS IN:

• Student demographics
• Financial aid factors
• Traditional vs Non-traditional status
• Full-time/Part-time status
• Major/Program pursued
• Credential pursued
• Online vs. On-ground vs. Other modalities of course taking
• Prior academic history

PARTICIPATION IN A COLLABORATIVE COMMUNITY OF EXPERTS: PAR Framework members represent an engaged community of forward thinking institutions, each with perspective and practices around student retention, progression, and completion. PAR membership provides institutional leaders with the models, technical support, services, and guidance needed to evaluate techniques, tools and solutions used to improve student outcomes. PAR provides the collaborative environment needed to support policy implementation and effective practice guidance necessary to capitalize on the power of data-driven decision-making for improving student success in reliable, repeatable, scalable ways.

 For more information about joining PAR, please contact us at [email protected].

 

Karen Swan and the Amazing Multi-institutional PAR Student Success Matrix

Back in June 2013 the PAR Framework openly published an intervention assessment framework called the Student Success Matrix to help PAR members figure out how to respond when confronted with predictions of which students were likely to drop out. "What should we do to respond to these predictions showing our students at risk?" our members asked.  At that time Dr. Karen Swan, Stukel Distinguished Professor of Educational Leadership at the University of Illinois Springfield, reminded us that there are already lots and lots and lots of things that academic advisors, faculty and other student success professional already do to help students succeed in school. Going back through the professional literature, she and Dr. Peter Shea, SUNY-Albany, helped us find models of student engagement and participation relevant to our work at hand.

Karen proposed a framework to map interventions using predictor behaviors of success and points in the academic completion cycle.  Over the past year Karen and Peter have worked closely with Sandy Daston, PAR Framework Director of Student Success and the entire PAR SSMx working group (Hae Okimoto, University of Hawaii System; Jennifer Freed, Dan Huston, and Fermin Ornelas, Rio Salado College; David Shulman, Broward College; Denise Nadasen, Jack Neill and Karen Vignare, UMUC; Patsy Moskal and DeLayne Priest, University of Central Florida; Michelle Wiley, Penn State World Campus; Mysti Reneau, Community Colleges of Spokane) to refine processes and tools for promoting intervention measurement. This group of experienced educational practitioners and decisionmaker has helped determine what can actually be done for student at risk - even down to knowing how to support those 37 students that are most likely to fail Math 101 this semester. 

PAR institutions have begun the process of inventorying, tagging, describing measuring and evaluating literally hundreds of interventions currenty in use on campuses to help student succeed. Thanks to the SSMx we have accelerated the process of connecting predictions of risk to the interventions that will mitigate risk. 

This post is to acknowledge Karen's leadership and contributions to PAR's thinking about using predictions as cataysts for taking action, and for linking interventions to predictions to underscore the importance of linking predictions to action.

 

When I realized how easy is was to get started with learning analytics, I laughed out loud

I've always wanted to write a headline like that.  

If you are like me, any link suggesting that learning analytics can be easy is worth a quick look. Even if only to sneer at the temerity of whoever would say that. So, thank you for being curious enough to click. Now that I have your attention...

Getting started with a learning analytics initiatve doesn't need to be hard. What's hard is sustaining a learning analytics initiative IF one is only doing something, anything, as a pet project, an exploration, or a trial run because someone gave you free software. 

To paraphrase the incomparable Tina Turner, we're gonna start this learning anaytics conversation nice and easy (don't worry, no dancing required). Here we go. Five quick easy steps to get started:

(1) Start with a real problem to solve. Scope your problem statement so that you maximize your probabiities of  success. Do you have market conditions to address, sneaking suspicions that there might be trouble in River City that you'd better find out about before someone else does? Maybe places where you are pretty sure you can do a better job? You might want to think of something that will help you test the waters of change without putting yourself at risk of total humiliation if it doesn't work.

(2) Figure out who needs to be involved in creating the business case to get this problem solved. How much of a priority is solving the problem you identified going to be for others in your organization? Who need to be engaged and on-board to get this done? 

(3) Determine what you need to answer questions related to solving this problem. What information do you need? Who owns it? Where does it live? How are you going to secure it? How will it be analyzed? Who can see it, work with it, publish it? How will results be used? How long will the results live, and who will have access?

(4) Determine the necessary project talent to make sure you can get the work done. It's not just about having access to people who understand how to work with data. It's about having the right team in place across the board who can execute to plan and deliver results, and making sure that you have planned for enough of their time to be effective. How will you find them? What will you need to attract their attention and keep them engaged?

(5) Develop a bottom-up budget to see if you can afford to do what you want to do. Odds are, you will need more money than you have on hand. (Well... odds are that you will end up needing more money than you think you do, too, but that's a conversation for a later time). So, where might you find the funding to do what you want to do? Who else might be interested in getting involved in your great adventure?

Obviously each one of these steps can take on a life of its own once you get going, but as a place to begin these are a great way to get your arms around your big idea and see if its possible to move ahead.

Now that Predictive Analytics are Hotter than Ever

Now that predictive analytics are hotter than ever in higher education, it's probably a good idea to review a few fundamentals for those who are just starting to sip from the cup of possibilities.

WHY PREDICTIVES? Predictive analytics have been used in a wide variety of settings, including higher education, to manage finances, inventory, operations, assets and resources. Increasingly, higher ed institutions are turning to business intelligence for enrollment management and student recruitment. The next great wave for predictive analytics adoption in higher education is focusing on institutional performance outcomes and individualized student success. 

WHY COLLEGES NEED TO CARE: The more that colleges and universities are being held accountable for achieving specific performance outcomes (e.g., improved student retention, better completion rates for college students), the more predictive analytics will be touted as a solution for anticipating risks likely to be encountered while trying to achieve those outcomes. 

SIMPLY KNOWING WHO IS AT RISK ISN'T ENOUGH. Knowing how to mitigate risks and how different students can be better served with targeted interventions and support makes predictions actionable. Predictions without action don't really matter very much to anyone.

BEWARE THE BRIGHT AND SHINY: It's going to be a little bit like the early days of the first dot.com out there, with lots of  solutions available and even more promises about all the great things that those interventions and solutions can do (!!!). How will you know who to believe? How will you know what's right for you? You owe it to yourself to be informed.

THERE IS NO SINGLE "RIGHT WAY" TO CONDUCT PREDICTIVE ANALYSES. There are many techniques used to conduct predictive analyses. Typically one chooses the technique or techniques likely to yield results for the kinds of predictions one wants to make.

For example, if you are interested in predicting a discrete attribute you might be likely to use techniques including Logistic Regression, Decision Trees (CHAID, CART, Random Forest), Naïve Bayes, Support Vector Machine, Survival Analysis or Neural Networks. If you were more interested in predicting a continuous attribute, you'd look toward techniques such as Multiple Linear Regression, Time series or Decision Trees (CHAID, CART, Random Forest). If you were most interested in common groups, you might consider Hierarchical k-nearest neighbors, Neural Networks or Decision Trees (CHAID, CART, Random Forest).

Selecting the right techniques used for conducting predictive analyses have a lot to do with knowing the questions that the predictions will help answer, or the performance problems that are likely to be solved. Speaking of which...

START WITH A PROBLEM TO SOLVE: Sorry to burst your bubble but predictive analytics don't work quite like Magic 8-Ball. Not even Hadoop technologies do that. Answers simply do not emerge fully formed from the mists of your analytic techniques.  It's easier to focus on solving a problem (e.g. what causes students to drop out? Are these causes common in all settings?) or finding a new opportunity (e.g. what motivates students to complete courses faster?). Your problem statements and queries will help you focus on finding data sources and selecting techniques for analyses that are likely to reveal the patterns you seek.