CCVis: Visual Analytics of Student Online Learning Behaviors Using Course Clickstream Data

Abstract: As more and more college classrooms utilize online platforms to facilitate teaching and learning activities, analyzing student online behaviors becomes increasingly important for instructors to effectively monitor and manage student progress and performance. In this paper, we present CCVis, a visual analytics tool for analyzing the course clickstream data and exploring student online learning behaviors. Targeting a large college introductory course with over two thousand student enrollments, our goal is to investigate student behavior patterns and discover the possible relationships between student clickstream behaviors and their course performance. We employ higher-order network and structural identity classification to enable visual analytics of behavior patterns from the massive clickstream data. CCVis includes four coordinated views (the behavior pattern, behavior breakdown, clickstream comparative, and grade distribution views) for user interaction and exploration. We demonstrate the effectiveness of CCVis through case studies along with an ad-hoc expert evaluation.  Finally, we discuss the limitation and extension of this work.

Celeste, M., Gronda E., Yang, Y., Tao, J., Wang, C., Duan, X., Ambrose, G., Abbott, K., Miller, P. (2018) CCVis: Visual Analytics of Student Online Learning Behaviors Using Course Clickstream Data. IS&T International Symposium on Electronic Imaging Conference. [Click here for the paper]

Note this paper was awarded the Kostas Pantazos Memorial Award for Outstanding Paper in Visualization and Data Analysis

Kaneb, OIT, and FYS Team win 2018 Apereo Teaching And Learning Award (ATLAS)

3 min trailer about the project

A cross-campus team composed of faculty and staff from Kaneb, OIT, and FYS were awarded the 2018 Apereo Teaching And Learning Award (ATLAS). This international award recognizes innovation and excellence in the use of Apereo tools (Sakai), which enhance teaching, academic collaboration, student engagement, and learning. The major goals of the award include:
-Promote excellent pedagogy and innovation in teaching and learning;
-Create a community of educators who want to share teaching and learning practices;
-Encourage greater faculty involvement in the Apereo community.

The Apereo Foundation ( is a non-profit organization, with members drawn from higher education on four continents. It’s core mission is to provide a framework to assist and facilitate educational organizations to work together to foster, develop, and sustain open source technologies and innovation to support learning, teaching, and research. Apereo also provides a common organizational umbrella for higher education open source software projects and communities. Many of Notre Dame’s peer organizations are part of Apereo. Those include Duke, Yale, Stanford, Brown, Princeton, and others.

Each year, the Foundation selects nominations for the ATLAS award (Apereo Teaching and Learning Award) from the open source community that demonstrate innovative teaching and learning using an open source learning tool. Those awards include recognition at the annual Apereo international conference and funding for those team members to attend the conference and present on the winning nomination.

Our proposal was on the Moreau First Year Experience which is a required, two-semester sequence that helps first-year students make a meaningful transition to collegiate life at Notre Dame by integrating their academic, co-curricular, and residential experiences. Apereo tools form the digital framework of our courses. Our open-source learning management system, Sakai, contains all resource materials for both students and instructors as well as weekly reading/viewings and short reflections; a real-time gradebook; and a seamless interface with midterm and final ePortfolios assignments. Apereo tools ensure an innovative and transparent learning environment for our students, instructors and course administrators. We also worked with Digication, our ePortfolio provider, to create an LTI that would connect Digication to Sakai.  That work was funded by Digication and we assisted with testing. The LTI was released by Digication for use by the Apereo community.

For more information:
-Click here for the announcement and to learn more about the award.
-Click here for our award submission
-Click here to read the story on the Academic Technology @ Notre Dame Blog

The interdisciplinary cross-campus team included:

  • Kevin Abbott, Office of Information Technologies
  • Alex Ambrose, Kaneb Center for Teaching & Learning
  • Alison Lanski, Kaneb Center for Teaching & Learning
  • Laura Cira, Office of Information Technologies
  • Laura Gekeler, Office of Information Technologies
  • Kevin Barry, Kaneb Center for Teaching & Learning
  • Maureen Dawson, First Year of Studies
  • Xiaojing Duan, Office of Information Technologies
  • Patrick Miller, Office of Information Technologies
  • Trunojoyo Anggara, First Year of Studies
  • Chris Clark, Kaneb Center for Teaching & Learning
  • Paul Turner, Office of Information Technologies
  • Munira Syed, Interdisciplinary Center for Network Science & Applications
  • Nitesh Chawla, Interdisciplinary Center for Network Science & Applications


Abbot, Kevin, Dawson, Maureen, Ambrose, G. Alex (2018) “How to Use Sakai and the Open Learning Record Warehouse with Learning Analytics to Empower First-Year Students to Thrive”  Apereo Teaching & Learning Awards (ATLAS), Open Apereo Conference, Montreal, Canada.

For more information:
-To hear the full presentation you can watch the recorded webinar on YouTube here
-To view the slide deck from our presentation see the embedded google slides below

Award Presentation Abstract

At Notre Dame, our goal is for all our freshmen students to thrive. According to U.S. News & World Report, Notre Dame ranks in the top 10 of national universities for freshman retention with a 98% retention rate. You might be wondering why we would be concerned with our students thriving since 98% of our freshman class succeeds. Our challenge is to engage those struggling 2% early enough to take action.

Our First Year Experience course was designed to help our students reach their highest potential. We maximized Sakai for the overarching structure of the course and Open Learning Record Warehouse for gathering data. These ensured that all resources were available to students, at no cost, throughout the semester; that student work would be evaluated promptly and fairly; that course design and student progress were transparent; and that data was gathered for learning analytics and research.

This session will walk you through how we designed and implemented the course, connected the data to the Learning Record Warehouse, how we used predictive analytics to move 83% of students identified as underperforming to thriving, and what is the best and earliest predictor of student success.


Poster & Paper: The ABC of MOOCs: Affect and its inter-play with behavior and cognition

Shazia Afzal, Bikram Sengupta, Munira Syed, Nitesh Chawla, G. Alex Ambrose, and Malolan Chetlur. “The ABC of MOOCs: Affect and its inter-play with behavior and cognition.” In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 279-284. IEEE, 2017.

Click here to zoom in and download the poster.

Abstract—We report on a study of affective states of learners in a Massive Open Online Course (MOOC) and the inter-play of Affect, Behavior and Cognition at various stages of the course. Affect is measured through a series of self-reports from learners at strategic time posts during the period of study. Behavior is characterized in terms of a learners’ engagement, interactivity, impatience and reflectivity, which constitute a set of novel high-level features derived from the clickstream of learner interactions. Cognition is evaluated from the performance of learners on assessments that are part of the course.We discover that learners in the MOOC experience multiple
as well as mixed emotions as they go through the course, which we handle using the psychological dimensions of arousal and valence. This results in a set of emotional quadrants, whose co-occurrence analysis reveals a strong association with
cognition and specific behavioral characteristics demonstrated by the learner. These results advance our understanding of the experience of MOOC learners to a more holistic level across the key dimensions of affect, behavior and cognition. They
also have important implications for the design of the next generation MOOCs that can potentially leverage affect and behavior-aware interventions to drive greater personalization and eventually, improved learning outcomes.


Liberate Learning through Next Generation Assessment — Closing Plenary

Closing Plenary of the AAC&U’s 9th Annual Forum on Digital Learning and ePortfolios

Liberate Learning through Next Generation Assessment

The ability to authentically capture and assess student opinions and growth has always been a challenge. Polls, surveys, and focus groups are some of the most common ways for administrators to gather assessment data at an institutional level; however, these indirect sources only serve as proxy indicators of student voices and experiences. And while ePortfolios may provide better and more authentic direct assessment, they are usually not faster or easier. How can we merge the efficiency of the forms, polls, and surveys with the power of ePortfolios to have a faster, better, and a deeper look into our students’ expectations? We will provide an overview of our current text mining assessment methods for over 2000 students ePortfolio reflections given specific prompts. These methods will provide a counter-narrative to indirect surveys that liberate all of our students’ unique voices and allow us to better align their aspirations in the arena of higher education. And answer the questions…What does assessment in higher education actually do to the American dream? How can we move beyond efficient assessment to more effective assessment thereby liberating the learning and the learners?

G. Alex Ambrose, Program Director of ePortfolio Assessment, University of Notre Dame
Trunojoyo (Atun) Anggara, Academic Advisor and Educational Data Assessment Postdoctoral Fellow, University of Notre Dame

Access Slides: 
Liberate Learning through Next Generation Assessment -AACU 2018 Closing Plenary 

Ambrose, G. Alex, Anggara, Trunojoyo (2018) “Liberate Learning Through Next Generation Assessment” Association of American Colleges & Universities, 9th Annual Forum on Digital Learning and ePortfolios Closing Plenary. Washington, D.C

Pushing the Frontiers of the Next Generation Digital Learning Environment

Ambrose, G. Alex, Abbott, Kevin, Lanski, Alison (2017) “Under the Hood of a Next Generation Digital Learning Environment in Progress” Educause Review.

Key Takeaways

*A first-year experience course at the University of Notre Dame offered the opportunity to develop and test a next generation digital learning environment.

*As a guiding framework for the digital architecture of the new FYE course, the design team followed the five dimensions of an NGDLE described in an ELI report to close the gaps between current learning management tools and digital learning environments.

*The FYE course required innovating beyond the supplemental course-level learning management system to create a more holistic NGDLE in order to fully support the teaching, learning, and student success missions of the program.

*Because they used an open-source LMS, they had a flexible hub with a campus support staff that could quickly, inexpensively, and safely begin to innovate at scale without high vendor cost or long development times.


Digital Badging a Conference on Digital Portfolios & Badges

On May 11-12, 2016 The REAL Design Lab in the Kaneb Center for Teaching & Learning at the University of Notre Dame hosted the AAEEBL 2016 Midwest Regional Conference with a theme about how digital portfolios and badges could work together. To see the conference details, program schedule, and published slides click here.

One of the big problems we have seen so far with early conference badging is that they turn out to be what Chris Clark calls “warm-body” badges. These badges don’t have any meaning or value besides “I have been there and got the t-shirt.”

The conference committee and badge design team had a goal to model good quality and practice with conference badging. In addition, we wanted to recognize and motivate presenters to “micro-publish” their presentations so we could share their work with a larger community that could not attend the conference. So we created the AAEEBL Midwest Regional Conference Presenter Badge.


Click here to see verified page on Credly.

To read the presenter reflections and view their slides you can either click on the evidence link in their digital badges above or see our badge directory of presentation materials here

To look under the hood and see how we pulled this off see the concept map below:


Click to zoom in

This hack of using LinkedIn Pulse (blog post) turned out be an effective platform for participants to share their public reflections, publish their slides (using Slideshare which was just bought out by LinkedIn) and also gave them a place to showcase their earned digital badge.

ND Issues (& Studies) 5 different Badges in 3 MOOCs issued to 233 global learners

Over the last year, The University of Notre Dame’s Kaneb Center for Teaching & Learning and the Office of Digital Learning issued and designed 5 different (Credly) Open Badges in 3 (edX) MOOCs to 233 learners around the globe.

Click on image to zoom in

Click on image to zoom in

To read a white paper on examining digital badge impact on learners’ profiles, performance, & perceptions see: “Notre Dame Issues 151 Digital Badges in an edX MOOC

To view some slides about a study on the second MOOC Digital Badge see: “AAEEBL Conference Presentation on Role of Digital Badges in a Massive Open Online Course (MOOC)

To hear a story from our third MOOC Digital Badge experience see: “What Alaa Taught us about Open Learning Evidence (in a MOOC with a Digital Portfolio & Badge)

Twitter 140 character sum:
@NotreDame studies 5 different @Credly #Open Badges in 3 @edX #MOOCs issued to 233 global learners

Webinar: Pairing Digital Portfolios & Badges to Recognize Informal Learning

Ambrose (2015) “Pairing Digital Portfolios & Badges to Recognize Informal Learning” The Badge Alliance’s Open Badges in Higher Ed, Webinar

G. Alex Ambrose will discuss why digital badges need ePortfolios and why ePortfolios need digital badges. “ePortfolios with Evidenced-Based Badges” (E2B2) at Notre Dame, which won a 2015 Campus Technology Innovator Award, is this first true integration of ePortfolios (Digication) and digital badges (Credly) offers evidence of competencies gained from co-curricular learning experiences on a collegiate campus. To date, 9 different badges have been created and issued to over 100 students. Using a few use cases, Alex with share his research on digital badge design, perceptions, and impact.

Audio Recording (8/4/2015):
Backchannel Notes:


Keynote: ePortfolios @ Scale and Beyond with Badges & Analytics

Ambrose, G. Alex (2015) “ePortfolios @ Scale and Beyond with Badges & Analytics” Association of Authentic, Experiential Evidence-Based Learning (AAEEBL) Annual Conference Closing Keynote, Boston, MA.

For the slide deck see:

For the Digital Handout & Backchannel see: