Publications

Understanding Computing Education

This set of studies explored computing education as a field from different perspectives. This project is ongoing. The current angle we choose is “replication”.

A Systematic Investigation of Replications in Computing Education Research.
Qiang Hao, David H. Smith, IV, Naitra Iriumi, Michail Tsikerdekis, and Andrew J. Ko
ACM Transactions on Computing Education. , 19(4), 1-18.

Abstract:
As the societal demands for application and knowledge in computer science (CS) increase, CS student enrollment keeps growing rapidly around the world. By continuously improving the efficacy of computing education and providing guidelines for learning and teaching practice, computing education research plays a vital role in addressing both educational and societal challenges that emerge from the growth of CS students. Given the significant role of computing education research, it is important to ensure the reliability of studies in this field. The extent to which studies can be replicated in a field is one of the most important standards for reliability. Different fields have paid increasing attention to the replication rates of their studies, but the replication rate of computing education was never systematically studied. To fill this gap, this study investigated the replication rate of computing education between 2009 and 2018. We examined 2,269 published studies from three major conferences and two major journals in computing education, and found that the overall replication rate of computing education was 2.38%. This study demonstrated the need for more replication studies in computing education and discussed how to encourage replication studies through research initiatives and policy making.

Automated Feedback

This set of studies explored automated feedback on student programming assignments. This project is ongoing.

Investigating the Essential of Meaningful Automated Formative Feedback for Programming Assignments.
Qiang Hao, Jack Wilson, Camille Ottaway, Naitra Iriumi, Kai Hicks and David H. Smith, IV
2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). Memphis, TN.

Abstract:
This study investigated the essential of meaningful automated feedback for programming assignments. Three different types of feedback were tested, including (a) What’s wrong - what test cases were testing and which failed, (b) Gap - comparisons between expected and actual outputs, and (c) Hint - hints on how to fix problems if test cases failed. 46 students taking a CS2 participated in this study. They were divided into three groups, and the feedback configurations for each group were different: (1) Group One - What’s wrong, (2) Group Two - What’s wrong + Gap, (3) Group Three - What’s wrong + Gap + Hint. This study found that simply knowing what failed did not help students sufficiently, and might stimulate system gaming behavior. Hints were not found to be impactful on student performance or their usage of automated feedback. Based on the findings, this study provides practical guidance on the design of automated feedback.

Quantifying Student Prior Computer Science Knowledge

This set of studies explored the measurements of student prior CS knowledge and their effects on student performance. This project is ongoing.

Quantifying the Effects of Prior Knowledge in Entry-Level Programming Courses
David H. Smith, IV, Qiang Hao, Filip Jagodzinski, Yan Liu, and Vishal Gupta
ACM 2019 Conference on Global Computing Education. Chendu, China.

Abstract:
Computer literacy and programming are being taught increasingly at the K-12 level with more students than ever matriculating in college with prior programming experience. Accurately assessing student programming skills acquired in high school can inform college faculty about the range of competencies in introductory programming courses. The tool predominantly-used for assessing past CS knowledge and skills is a survey, which lacks quantitative rigor. This study aims to (1) quantify the effects of prior knowledge in entry-level programming courses and (2) compare the different measurement approaches of student prior knowledge in programming, including surveys and aptitude tests. The results of this study reveal that a discrepancy exists between the results of surveys and aptitude tests. Consistent with prior survey studies, our survey results showed that the effects of student prior programming knowledge faded gradually during the course period. In contrast, the aptitude test results indicated that the effects of student prior knowledge did not weaken over time. The accuracy of both measurements and implications for instructors were further discussed.

Online Help seeking

This set of studies explored online help seeking from different angles in the context of computing education. This project is inactive now, but I am happy to answer any questions about our findings.

Automatic Identification of Ineffective Online Student Questions in Computing Education
Qiang Hao, April Galyardt, Bradley Barnes, Ewan Wright, Robert Maribe Branch
2018 IEEE Frontier in Education (FIE ’18). San Jose, CA.

Abstract:
This Research Full Paper explores automatic identification of ineffective learning questions in the context of large-scale computer science classes. The immediate and accurate identification of ineffective learning questions opens the door to possible automated facilitation on a large scale, such as alerting learners to revise questions and providing adaptive question revision suggestions. To achieve this, 983 questions were collected from a question & answer platform implemented by an introductory programming course over three semesters in a large research university in the Southeastern United States. Questions were firstly manually classified into three hierarchical categories: 1) learning-irrelevant questions, 2) effective learning-relevant questions, 3) ineffective learning-relevant questions. The inter-rater reliability of the manual classification (Cohen’s Kappa) was.88. Four different machine learning algorithms were then used to automatically classify the questions, including Naive Bayes Multinomial, Logistic Regression, Support Vector Machines, and Boosted Decision Tree. Both flat and single path strategies were explored, and the most effective algorithms under both strategies were identified and discussed. This study contributes to the automatic determination of learning question quality in computer science, and provides evidence for the feasibility of automated facilitation of online question & answer in large scale computer science classes.

Online help seeking in computer science education (Doctoral dissertation)
Qiang Hao
University of Georgia, 2017.

Predicting Computer Science Students’ Online Help-Seeking Tendencies
Qiang Hao, Bradley Barnes, Robert Maribe Branch, Ewan Wright
Knowledge Management & E-Learning, 9(1), 19-32, 2017.

Abstract:
This study investigated how computer science students seek help online in their learning and what factors predict their online help-seeking behaviors. Online help-seeking behaviors include online searching, asking teachers online for help, and asking peers online for help. 207 students from a large university in the southeastern United States participated in the study. It was revealed that computer science students tended to search online more frequently than ask people online for help. Five factors, including epistemological belief, interest, learning proficiency level, prior knowledge of the learning subject, and problem difficulty, were explored as potential predictors in this study. It was found that learning proficiency level and problem difficulty were significant predictors of three types of online help-seeking behaviors, and other factors influenced online help seeking to different extents. The study provides evidence to support that online searching should be considered as an integrated part of online help seeking, and gives guidelines for practice of facilitating online help seeking and future studies.

The Influence of Achievement Goals on Online Help Seeking of Computer Science Students
Qiang Hao, Bradley Barnes, Ewan Wright, Robert Maribe Branch
British Journal of Educational Technology, 48(6), 1273-1283, 2017.

Abstract:
This study investigated the online help‐seeking behaviors of computer science students with a focus on the effect of achievement goals. The online help‐seeking behaviors investigated were online searching, asking teachers online for help, and asking peers or unknown people online for help. One hundred and sixty‐five students studying computer science from a large research university in the south‐eastern United States participated in the study. It was found that students searched online significantly more frequently than they asked people online for help. Contrary to prior findings on face‐to‐face help seeking, no achievement goals were found to be significant in predicting the tendencies of students to seek help online. These findings provide evidence to support the role of online searching as an integral part of online help seeking and demonstrate that research findings on face‐to‐face help seeking should not be assumed to be naturally extendable to online help seeking.

What Are the Most Important Predictors of Computer Science Students’ Online Help-Seeking Behaviors?
Qiang Hao, Ewan Wright, Bradley Barnes, Robert Maribe Branch
Computers in Human Behaviors, 62, 467-474, 2016.

Abstract:
This study investigated the online help‐seeking behaviors of computer science students with a focus on the effect of achievement goals. The online help‐seeking behaviors investigated were online searching, asking teachers online for help, and asking peers or unknown people online for help. One hundred and sixty‐five students studying computer science from a large research university in the south‐eastern United States participated in the study. It was found that students searched online significantly more frequently than they asked people online for help. Contrary to prior findings on face‐to‐face help seeking, no achievement goals were found to be significant in predicting the tendencies of students to seek help online. These findings provide evidence to support the role of online searching as an integral part of online help seeking and demonstrate that research findings on face‐to‐face help seeking should not be assumed to be naturally extendable to online help seeking.

Active Learning Environments in Computing Education

This set of studies explored the effects of active learning environments in computing education. This project is inactive now, but I am happy to answer any questions about our findings.

On the Effects of Active Learning Environments in Computing Education
Tyler Greer, Qiang Hao, Mengguo Jing, Bradley Barnes
2019 ACM Technical Symposium on Computer Science Education (SIGCSE ‘19). Minneapolis, MN.

Abstract:
This replication study aims at both quantifying the effects of active learning classrooms in introductory programming courses (CS1) and overcoming some design and methodological limits of prior studies on this topic. 156 students enrolled in three different sections of the same CS1 participated in this study. The three sections differed from each other either in terms of learning pedagogies (conventional lecture vs. peer instruction) or physical learning environments (lecture hall vs. active learning classroom). This study did not replicate the findings of prior studies on this topic. Instead, this study found that when learning pedagogies were controlled, learning environments did not have significant influences on student performance. On the other hand, learning pedagogies were found to have significant influences on student performance. When peer instruction is conducted other than conventional lecturing, students tended to have significantly better performance. Such findings highlight the importance of active learning in computing education, and the feasibility of conducting active learning in CS1 despite of physical environment constraints. Additionally, such findings emphasize the necessity of replication studies on the topic of active learning environments, and invite debates on the investment decisions in active learning classrooms.

Effects of Active Learning Environments and Instructional Methods in Computer Science Education
Qiang Hao, Bradley Barnes, Ewan Wright, Eunjung Kim
2018 ACM Technical Symposium on Computer Science Education (SIGCSE ‘18). Baltimore, MD.

Abstract:
This research investigated the impacts of active learning environments and instructional methods adapted to such environments on the academic performance of computer science students. Two consecutive studies involving a total of 267 novice students in the same course were conducted across two different semesters. The course was taught by the same instructor and set up with two different sections. One section was taught in a conventional lecture hall, while the other was taught in an active-learning classroom with adapted instructional methods. Active learning environments and the adapted instructional methods were found to have significantly positive effects on students’ learning outcomes. Fine-grained results grouped by major were discussed. The findings of this study demonstrate positive effects of active learning environments in computer science education, thereby adding to the literature on both computer science education and learning environments.

Feature Selection on Predicting Post-graduation Income

This set of studies explored the most important factors on predicting post-graduation income. This project is inactive now, but I am happy to answer any questions about our findings.

Feature Selection of Post-graduation Income of College Students in the United States
Ewan Wright, Qiang Hao, Khaled Rasheed, Yan Liu.
2018 International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation (SBP-BRiMS 2018). Washington, D.C.

Abstract:
This study investigated the most important attributes of the 6-year post-graduation income of college graduates who used financial aid during their time at college in the United States. The latest data released by the United States Department of Education was used. Specifically, 1,429 cohorts of graduates from three years (2001, 2003, and 2005) were included in the data analysis. Three attribute selection methods, including filter methods, forward selection, and Genetic Algorithm, were applied to the attribute selection from 30 relevant attributes. We discuss how higher numbers of students in a cohort who grew up in Zip code areas where over 25% of the population hold a Professional Degree was predictive of more college graduates being classified as High income.

Feature selection and classification of post-graduation income of college students in United States (Master’s thesis)
Qiang Hao
University of Georgia, 2017.