In this study we use element-level usage data that was collected from the online learning material of an university level introductory programming course for identification of areas-of-interest in the course material as well as for prediction of student learning outcomes. The data was collected in-situ using a JavaScript component embedded in the online learning material, which recorded which HTML elements were visible on the user's screen after each interaction (movement and click) and if the user's screen had been still for at least 2500 milliseconds. A visual analysis indicates that students spend large amounts of time on material sections that discuss special syntactic structures that they are unable to infer from previous experience. Overall, the analysis was able to identify areas of the online learning material that seem to be too long and in-depth for the concepts they are discussing, when the things the students have previously learned are taken into account. This high-level analysis also revealed that the time the students spent viewing an assignment's prompt was statistically significantly correlated with the perceived workload, difficulty and educational value of that same assignment. We observe that when partial correlations are considered, and multiple comparisons are corrected for, time spent with an assignment's prompt on the screen is no longer statistically significantly correlated with the three variables.
The same usage data was used to investigate whether material usage statistics can predict learning outcomes or identify strong and at-risk students. The results indicate that based on just three to four weeks of data, it is possible to identify strong and at-risk students with some accuracy. Furthermore, it seems possible to identify student programming assignment scores and total course scores with a somewhat high accuracy. Models based on material usage statistics also displayed some light predictive power in predicting student exam scores. It was also shown that the predictive powers of these models are not based solely on student effort or time-on-task.
All told, this thesis demonstrates that fine-grained online learning material usage data is feasible to collect and useful in understanding both the students and the learning material. The results suggest that very simple and almost entirely domain-independent data sources can be used to predict student performance to a relatively large degree, suggesting that a combination of such simple domain-independent metrics could match highly domain dependent and more complex metrics in predictive power, giving raise to more widely usable educational analytics tools.
