Jiving with LLMs: Assessing First Year Students’ Computer Programming Self Efficacy After Reading Code With LLMs
Article Sidebar
Main Article Content
Abstract
This study investigated the impact of leveraging generative artificial intelligence (GenAI) to assist 1st-year engineering and computer science (CS) students in reading code in a new (to them) language. Students were asked to comment code in FORTRAN. They were then asked to run the code through ChatGPT-4.0 for its comments and reflect on what they learned from the experience. Participants completed survey items from Ramalingam and Wiedenbeck’s Computer Programming Self-Efficacy Scale (CPSES) prior to and after the intervention. Additional open-ended reflective (qualitative) questions were added to the quantitative questions in the postintervention questionnaire. This study documents increases in self-efficacy for programming independence and persistence (Factor 1 of the CPSES) as well as for complex programming tasks (Factor 2) after students used ChatGPT for generating code explanations. Both CS and engineering students showed improvements in programming independence and persistence; but only engineers showed significant improvements in their confidence regarding complex programming tasks. Men experienced a significant increase in self-efficacy on Factor 1 while women experienced a significant increase on Factor 2. The qualitative data point to an increase in student understanding of the new code and suggest that although students may be more likely to use GenAI for assistance as they progress through programming courses, guiding students in using GenAI to understand code may shift students’ intent away from using GenAI to write code for them. Thus, we recommend that programming faculty instruct students how to interact with GenAI.
Downloads
Article Details
- Authors retain copyright and grant the Journal of Teaching and Learning with Technology (JoTLT) right of first publication with the work simultaneously licensed under a Creative Commons Attribution License, (CC-BY) 4.0 International, allowing others to share the work with proper acknowledgement and citation of the work's authorship and initial publication in JoTLT.
- Authors are able to enter separate, additional contractual agreements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in JoTLT.
- In pursuit of manuscripts of the highest quality, multiple opportunities for mentoring, and greater reach and citation of JoTLT publications, JoTLT encourages authors to share their drafts to seek feedback from relevant communities unless the manuscript is already under review or in the publication queue after being accepted. In other words, to be eligible for publication in JoTLT, manuscripts should not be shared publicly (e.g., online), while under review (after being initially submitted, or after being revised and resubmitted for reconsideration), or upon notice of acceptance and before publication. Once published, authors are strongly encouraged to share the published version widely, with an acknowledgement of its initial publication in JoTLT.