eLearning Roadtrip

Since You've Asked: More About the PAR Framework

The December 15, 2011 Chronicle of Higher Education's Wired Campus blog by Marc Parry shared news about a groundbreaking project we have taken on at WCET.  It describes our efforts to create a single, federated database of de-identified student records contributed by multiple universities and colleges. Six forward-thinking WCET member institutions have worked together this past six months to see if it was possible to analyze variables common across these institutions to look for patterns that can inform us about student loss and momentum. I'm pleased to say that we have succeeded. Baby steps. But still.

We've received a lot of questions. Here are a few answers. There will be more.

This is a multi-phased project. We are concluding our work on Phase 1. The best is yet to be.

Our first phase focused on demonstrating that it is posssible to get multiple postsecondary institutions here in the US to contribute their data to create a single unified data set. We are now applying a variety of descriptive, inferential and predictive analyses to look for patterns showing variables likely to affect student success. 

Phase 1 = Proof of Concept. Creating a single federated data set from the de-identified student records from six unique institutions has never been done before. 

We've started with 6 schools. We are adding additional schools as we continue our efforts to build out the federated data set. While 640,000 students records and 3.1 million course records are a lot more than most of us are used to working with, it is still just a sample from 6 schools. 

We've started with 34 variables that are common across each of the institutional partners. Much of our current work has focused on establishing common definitions of these variables to ensure that comparisons and aggregations are valid, reliable and repeatable. There will be more variables.

We are currently focused on online programs. Just about every single transaction associated with an online program is digitally captured and stored somewhere. It seemed the obvious, self-evident place to begin.

We've focused on pattern strategy exploration, not null hypothesis testing (at least not in the conventional sense). 

We're really not trying to answer research questions using big data analysis techniques. Not yet. We'll get there. First things first. The journey of a thousand miles begings with the first steps.

More to the point...if we do our work right, then perhaps some day YOU will be answering YOUR research questions using the federated data set using big data analysis techniques.

The team working on this effort brings expertise from institutional research, statistical analysis, quant and qual research, cognition and instruction, online learning, higher ed administration, educational public policy, project management, and commercial software development. They come from the public and for-profit sectors. Then are from community colleges and 4 year comprehensives. Several of our institutional partners are already award-winning, acknowledged practice leaders in the hot, hot, hot domain of learning analytics. These are some of the smartest, most dedicated, creative, committed people I know. 

None of us - not WCET, not the core team of experts, not any of our fabulous institutional partners - are making any kinds of reccommendations from our findings yet. You may have noticed that my quote in the Chronicle was about as non-specific as one can possibly be. But be assured, there are going to be some very interesting implications for practice and policy that will emerge as the new patterns we are finding are more sharply focused in the coming weeks and months. 

Yes of course we are going to share our findings. Sharing is a core WCET value. We exist to share. That's what Cooperatives do.

 

On Secret Handshakes and Making the World a Better Place with ID

I attended the 2011 annual conference of the Association for Educational Communications and Technology (AECT) in Jacksonville, FL this past week. AECT is a professional association focused on scholarship in the academic disciplines associated with instructional design. Its members are primarily college professors and their graduate students; members also include school librarians, media specialists, computer center directors, public television and publishing executives.

I spoke on the topic of Secret Handshakes of Instructional Design. My presentation was based on various blog posts I have written on this topic, as well as an essay published in the inaugural issue of the Journal of Applied Instructional Design on said same. It was intended as a gentle yet pointed look at this ID thing that we do. These may be the best of times for those who understand how to create experiences that inspire and enable learning, but our practice may be better served if we make our secret handshakes more inclusive, our practice less theoretical and abstract, and all our designs more actionable.

Here is the deck Download FINAL AECT Secret Handshakes for distrib I used in my presentation. I've removed copyrighted materials. All photos used in the deck come from iStockphoto. This presentation deck is distributed under a Creative Commons Attribution Share Alike, Non-commercial license.

Quick reflections - It was very nice to be back among the scholars. I enjoyed seeing former students and associates. It was great to talk with old friends, some I've known since my graduate school days. It was great to meet some new friends. I was inspired by the big ideas.

And yet, I was reminded that, regardless of our epistemological roots, our professional training, the number of or lack of our graduate degrees or the places where we work, many of us call ourselves IDs. And many of us in this wild and wacky ID practice universe continue to have a hard time recognizing one another. We come at this practice from diferent and various points in the ID value chain.

I was also reminded that big ideas and great intentions, while essential ingredients in successful recipes for change, are rarely sufficient for bringing about the kinds of systemic changes to culture and practice that are required to disrupt and transform.

Education transformation really is a team sport. Old ideas don't just fade away. Old ideas can be stubborn and relentless. Systemic change calls for many contributors. There are many variables to address, much practice and dedication required to develop expert skills. Transformation demands a clear understanding of what is to be changed, AND the will to see changed implemented. And with all due respect to everyone who claims the ID badge...We simply can't afford to all think the same way.

For real change - true transformation - we need all ID hands on deck. Scholars, analysts, artists, technologists, evaluators, and managers alike. Big thinkers. Expert practitioners. Skilled producers. People who can translate big ideas into actionable strategies and tactics driving real results about things that matter to the communities and stakeholders we purport to serve.

Secret handshakes DO help us recognize each other, help us evaluate what we and our fellow practitioners know, and what we don't. Maybe we DO need to make these "know 'em when you see 'em" attributes of our ID secret handshakes more explicit.  Maybe we really can use these secret handshakes as a conceptual checklist to find more of the expert practitioners who share ideas, information and practices that are similar or complementary to our own.

When you get right down to it, perhaps the most notable secret handshake all IDs share is the passion for finding solutions that make the world a better place for learning. And that is a great place to begin.

 

Lessons Revisited: RADAR, Pearl Harbor and Paying Attention to Evidence

One of our Sage Road Solutions LLC clients, the Western Interstate Commission for Higher Education, is holding its biannual meeting at the BYU Hawaii campus on the island of Oahu this week. Having spent the past several days in Denver hosting the annual meeting of the WICHE Cooperative for Educational Telecommunications, known as WCET), I'm still running on Mountain time. I woke up early, so I decided to take advantage of being in this glorious location to take a walk along the beach.

About 200 yards from the hotel an historical marker notes the location of the Opana mobile RADAR site. The story accompanying the marker offers an interesting object lesson for those of us trying to bring innovative applications of technology to the learning technology metaverse.

According to Wikipedia the term RADAR was coined in 1940 by the U.S. Navy standing for radio detection and ranging. It was brand new technology, showing enough promise that pilot programs were launched to test its utility. In 1941, a number of mobile radar trucks were deployed on Oahu to keep an eye on the skies near Hickam Field.

 

Early on the morning of December 7, 1941, two soldiers stationed at the mobile radar site near Opana Point saw electronic blips of what looked like many planes approaching the island. Reporting their findings, they were told that what they were seeing were *probably* planes expected from California, and not to worry about it.  Consequently this information was never reported to those in command. Shortly thereafter, the Japanese planes detected by the RADAR bombed Hickam Field and Pearl Harbor, destroying US ships and planes, leading to the US's declaration of war on Japan and igniting US involvement in World War II.

The historical information was certainly interesting in its own right. The narrative that accompanied the marker, however, was particularly striking:

"The failure to warn the Army or Navy command in Hawaii was not a failure of the technology as much as it was a failure of organization. While the technology of RADAR functioned as intended and detected the incoming planes, there was no way to accurately assess the information and communicate this knowledge to those in command." 

So, let's fast forward to October, 2011. On Thursday of this past week the WCET team working on our Predictive Analytics Reporting (PAR) Framework project revealed some of the first findings from our ground-breaking big data project. At the risk of sounding hyperbolic, the descriptive, inferential and predictive techniques being applied to this massive data set are providing some eye-opening perspectives on variables affecting student loss and momentum. As noted in a Forbes.com online article featuring Phil Ice, our project's principal investigator (who, by the way, really hates being called "the other Dr. Phil"...so please don't) these results are going to rock the way that educational decision-makers think about everything from college admissions, student services, technology investments and even "best practices" for online learning. 

Still, I have a feeling that these "big data" results will go through a period of being pooh-poohed by decision-makers who will have a hard time letting go of null hypothesis research methods. Traditional researchers' disdain for the so-called data mining techniques associated with pattern strategy and business intelligence used in other sectors are likely to get in the way. For many traditional researchers, educational data mining is a little bit like a stupid pet trick - even though these same folks really seem to like the convenience of Amazon shopping recomendations, Netflix movie recommendations and Pandora music recommendations.

I can't help but wonder if the findings about student loss and momentum that the cracker-jack research crew are sharing will be disregarded very much in the same way that Army and Navy command dismissed the evidence that Pearl Harbor was in harm's way because the techniques are too new to be taken seriously. I just hope that we can figure out how to help people pay attention to evidence before it is too late.

 

Big Data Comes to US Higher Education

WCET, the WICHE Cooperative for Educational Technologies, today announced the successful federation of datasets from the six institutions participating in the PAR (Predictive Analytics Reporting) Framework proof of concept project, just four months after the project’s early June launch. The goal of the PAR Framework is to identify variables that influence student retention and progression and to determine the impact of various demographic data on factors influencing loss and momentum. The data will be used to explore patterns that emerge when the datasets from considerably different institutions are analyzed as a single, unified sample. 

The dataset includes over 640,000 anonymized student records and over 3 million course level records, focusing on 33 common variables. The institutions represent public/private, two-year/four-year, and publicly-funded and proprietary institutions. 

“We have been using analytics as a tool for improving student retention and progression for several years,” said Dr. Phil Ice, vice president, research & development, American Public University System, and principal investigator of the PAR Framework Project. “Unifying records from multiple schools has the potential to expose generalizable patterns that can provide guidance to institutions across the board, regardless of their internal level of analytics expertise.” Ice continued, “In higher education, no predictive analytics project has been able to use a multi-institutional sample of this size. As a researcher, having a sample size of this number changes everything.” 

The WCET member institutions engaged in the PAR Framework project include American Public University System, Colorado Community College System, Rio Salado College, University of Hawaii System, University of Illinois Springfield, and the University of Phoenix. Each institution obtained IRB (Institutional Review Board) approval. All data was anonymized and encrypted to ensure that no data are personally identifiable. 

“We had been warned that we could never find six schools willing to collaborate on something as sensitive as an institutional student and course level data federation project,” noted Ellen Wagner, WCET Executive Director. “It’s exciting to see how forward-thinking WCET member institutions are bringing the power of “big data” to post-secondary education in the US. The magnitude of the data set unprecedented and the diversity of the participating institutions will provide us granular insight into similarities and differences across higher education institutions in the for profit, public and community college segments.” 

A summary of first findings are scheduled for presentations at the 23rd Annual WCET Conference in Denver, Colorado October 27th (https://wcetconference.wiche.edu/).

Thought You Might be Interested...

...in some of the thoughts I shared with my WCET crew today about learning analytics and lessons we can learn from Moneyball. It's an extension of my eLearning Roadtrip post on the same general topic.

I suppose this gives you an idea of what is top-of-mind for me these days. Well, that and wanting to raise the bar on online learning excellence. Simple pleasures.

If you are interested in taking a look at my WCET post, please click here. 

While there, please take a look around at what the fine folks at the WICHE Cooperative for Educational Technologies are doing these days. If you happen to care about accelerating the adoption of learning technologies innovation in practice you might find that WCET is engaged in work that matters to you.

As a matter of fact....WCET's Annual Meeting is being held in Denver October 26-29. We will be sharing some of the preliminary results from our Predictive Analytics Reporting Framework, along with lots of other information about learning analytics and other emerging topics of interest to online learning and learning technology professionals. Might be a great time to come see what the WCET buzz is all about.

WCET Receives Grant for Groundbreaking Higher Education Analytic Research

I am pleased to share the following good news.

WICHE, the Western Interstate Commission for Higher Education, announced today that WCET, the WICHE Cooperative for Educational Technologies, has been awarded a $1,000,000 grant from the Bill & Melinda Gates Foundation to fund an initiative to unify student data from six U.S. institutions and demonstrate the effective use of predictive analytic methods for improving student outcomes. The goal is to identify variables that influence student retention and progression, and guide decision-making that improves postsecondary student completion in the U.S.

This grant for the Predictive Analytics Reporting (PAR) Framework will aggregate data representing more than 400,000 student records from across six WCET member institutions: American Public University System, Colorado Community College System, Rio Salado College, University of Hawaii System, University of Illinois Springfield, and the University of Phoenix. Each participating institution has been exploring or implementing descriptive, inferential or predictive analytics projects on their own student data; the PAR Framework expands on this work through exploration of patterns that can be derived when the six institutional datasets are considered as a single, unified sample.

The principal investigator is Phil Ice, Ed.D., American Public University System (APUS). The project will incorporate many of the best practices and award-winning predictive analytical methods for which Phil has been recognized by the Sloan Consortium and for which project partner Dr. Vernon Smith, Rio Salado College, has won the prestigious WCET Outstanding Work (WOW) award. WCET has appointed Beth Davis, WCET, to serve as the project director.

The WICHE Cooperative for Educational Technologies (WCET) is a cooperative, membership-driven, non-profit provider of solutions and services that accelerate the adoption of effective practices and policies, advancing excellence in technology-enhanced teaching and learning in higher education. More information about WCET’s institutional membership resources, services and common interest groups can be found on here.

Lines are being drawn...over learning technology standards and specs?

Interesting article in today's Chronicle of Higher Education's Wired Campus about a growing shitstorm around learning technology standards.

Per Jeff Young, author of article in question: "The issue centers around a single line of the 53-page grant guidelines for the program, known officially as the Trade Adjustment Assistance Community College and Career Training Grants Program: “All online and technology-enabled courses developed under this [program] must be compliant with the latest version of SCORM (Sharable Content Object Reference Model).”

So what's the big whiff?

And then I read a little bit further and all became clear: Dr. Rob Abel, IMS Global CEO, is pitching an absolute hiss over this particular requirement. Why? Because his group supports a different standard.

There is a rich and storied pissing match history that has developed between Dr. Abel and the Advanced Distributed Learning Initiative of the US Department of Defense over the past several years, and I am very sorry to see that the stains are going to get all over this exciting new collaboration between the US Departments of ED and Labor. 

A bit of perspective -The federal government made a decision 11 years ago to develop SCORM as a specification to serve as a cost-saving criterion for RFP / vendor selection. ADL/DOD have been the keeper of SCORM on behalf of all the Feds. So it seems to me that the US Dept of Labor is well within their rights to adhere to their own government spec. Especially since the SCORM spec is built upon the standards coming from IMS Global, and other standards bodies, including AICC, IEEE-LTSC, and ARIADNE.

Hats off to Dr. Abel, he has done a great job persuading .edu that the IMS Global is the only standard that really matters. I mean that sincerely. IMS's Common Cartridge helps institutions get better value from LMS vendors and publishers. Even the most technically skilled educators don't want to worry about learning technology standards. The tap dances required to get things to interoperate in the old days of LMS ops were excruciating. Common Cartridge is a good thing, and he should be commended for his leadership on offering a solution to the big audacious higher ed  problem of content distribution and management.

But CCCT is a whole different initiative, one of several federal projects designed to force issues of openness precisely to have the different conversation. Because the IMS conversation only addresses part of the new opportunities. And the CCCT efforts are about tying education to training, jobs and development. Not the other way around.

(inserted note - SCORM dominates in corporate and government and  non-profit learning platform tech as an RFP "checklist" item. IMS dominates in higher ed. They are not mutually exclusive. They are different.)

It is pretty amazing to watch the lines getting drawn. Time to pull up a chair, grab some popcorn and get ready for some great pyrotechnics. 

There's a New Sheriff in (US Higher Education and Labor) Town

Dr. Darcy Hardy has been appointed to the Office of Assistant Secretary Jane Oats, Employment and Training Administration (ETA), United States Department of Labor. She will be supporting the recently announced Trade Adjustment Assistance Community College and Career Training (CCCT) Grant Program http://www.grants.gov ) that plans to award approximately $2 billion over the next four years.

She has taken a leave from her current position as Assistant Vice Provost for Technology Education Initiatives, University of Texas at San Antonio, Office of Information Technology, and has moved to Washington DC for her one year appointment.

I am very pleased (and relieved) that she will be able to continue on my WCET Executive Council during her time with the US Labor Department.

You many remember that, a few months back, Darcy and I had a bet over the World Series baseball championship. The loser of the bet was to present a webcast on a topic of the winner's choice. (There were a few other details of the bet, but we'll come back to those in a future post). Lucky me, I won the bet. I have asked Darcy to please plan for a WCET webcast in late February to tell us about the new CCCT Grant program she is heading up. 

Her first day on the new job is Monday January 24th. Let's give her a few weeks to get settled, then we'll get something scheduled. We'll be sure to announce the dates on the WCET website. And I'll post something here. 

January 24th is also Darcy's birthday. Isn't it great how the stars aligned for that to happen. Happy Birthday my friend!

Your online learning minions are here to serve. Sign us up.

 

 

What Dave Longanecker Said About Alaska

During the WCET Annual Meeting Thursday November 11, 2010 Luncheon presentation, during the session the Debate  between Dr. Dave Longanecker and Dr. Peter Smith on the topic of Rethinking the Academy in times of Great Change, a question about higher education cost was presented to the debaters from the audience.

Peter responded first, talking about the importance of choice, and that students benefit when there are a variety of options from subsidized to completely tuition driven. Then it was Dave's turn. And this is part of what Dave said:

"If you look at National Center for Higher Education Management data, there's the 250% difference in the cost per degree from one state to ...from the lowest cost state to the highest cost state. And it's not a quality...if you looked at the states you will see that quality is not the determining factor. The most expensive cost per degree state in the country is Alaska. I work with Alaska, I love Alaska, but I wouldn't put it as the most, as the best system in higher education in the country."

If you click here you will be redirected to the Sonic Foundry Mediasite archived recording of the Debate. Move the slider bar to the time-stamp at 45:45. (That's 45 minutes and 45 seconds into the session, less than two minutes before the entire session was adjourned). You can see and hear what Dave said for yourself.

Here is  the NCHEMS data that Dave was referencing when he made his comment about most expensive cost per degree state.

This slide was taken from a presentation that Dave had recently given to the CCCCIO conference in San Diego CA on October 28th. You can download the entire presentation here  Download Statistics from 101028 - CCCCIO Presentation on Quality Improvement Pretty interesting stuff about higher education that most of us typically don't see about cost, graduation, tuition and so on. 

His comments upset several of the WCET attendees from Alaska, who talked to me, talked to him, have written letters and fired off tweets. I know Dave has already apologized. It's not worth pissing people off. Especially when absolutely no harm was intended.

As this issue had been raised to public awareness through the #wcet10 tweet-stream I wanted to be sure people had access to the links to take a look for themselves, and to see some of the data that Dave was referencing.