Quantitative Analysis of the Evolving Student Experience During the Transition to Online Learning: Second-Language STEM Students

Abstract: The National University of Science Technology is an English-language institution in the Middle East offering degrees in Medicine, Pharmacy, and Engineering. We present the results of our studies of the evolving student learning experience during the 2019 coronavirus disease (COVID-19) lockdown restrictions. There was a discernible preference for synchronous, interactive learning. Mobile technology was more frequently used by our students than larger form factors. Platforms such as WhatsApp delivered much greater student engagement than our existing Learning Management System. Students learned at a slower rate using online material than we had anticipated with implications for assessment and progression.


Introduction
National University of Science and Technology (NU) in Oman, like many universities, was faced with a rapid switchover to online learning as a consequence of the 2019 coronavirus disease  epidemic. The STEM emphasis of our institution combined with second language students being taught by faculty for whom English is also a second language introduced further complexities. To ensure the effectiveness of our operations, the Quality Assurance Unit of the university undertook detailed surveys of our students and staff during this period. The surveys indicated that what we thought was needed, what the technologies were used for, and what the learning outcomes were all differed significantly from expectations.
The paper begins by introducing NU and the transition to online teaching required by our Ministry of Higher Education. The decisions we made to manage an immediate switch to online delivery are reviewed and discussed. Following on from this, we introduce the ways we monitored delivery, the methodologies employed, and the resultant analysis of our usage. The major findings are discussed together with their implications for teaching in the academic year 2020-21. Finally, some concluding remarks about future areas for investigation are made.

Background
NU was formed in 2018 as a merger of three existing colleges with histories dating back to 1996. NU operates under a license from the Ministry of Higher Education, Sultanate of Oman, and is recognized by the Oman Academic Accreditation Agency (OAAA). It has degree awarding powers, and where relevant, degrees also receive international program accreditation. We work closely with three academic affiliates: two in the United States (Universities of West Virginia and South Carolina) and one in the United Kingdom (Glasgow Caledonian University), as well as 22 other international collaborations for research and student exchange activities.
In 2020, NU has close to 4000 students and delivers MD, Masters, and Bachelors programs in Medicine, Pharmacy. and Engineering. As frequently found in the region, the majority of students take a preliminary year at our School of Foundation Studies, which eases the transition between an Arabic-language school system and an English-language university.
The majority of students speak English as a second language. The students are primarily (80%) Omani with the remainder coming from the region. Iraqi, Egyptian, Iranian, and Bahraini students account for around 50% of our international cohorts. The faculty originates mostly from Oman and South Asia, again with English as a second language.
Oman, like many parts of the world, has been affected by COVID-19. On March 15 th of 2020, all Higher Educational Institutions in Oman were instructed to move to an online teaching model immediately. We had to commence online operations the very next day. While all our colleges made regular use of Learning Management Systems (LMS), NU made little use of online teaching. There is a range of reasons for this, including: • the difficulty to digitize some of the subjects we taught, • regional concern about the probity of online qualifications, • conservatism amongst faculty members about online learning, and • worries about the ability of our students in rural areas to access the necessary technology.
Whatever the reasons, it was an overnight change, which required us to expeditiously devolve authority as to content and platform to individual colleges. Additionally, we needed to rely on our IT services to augment our infrastructure and provide the necessary equipment to faculty, and we needed our Quality Assurance (QA) and Teaching and Learning (T&L) functions to support faculty through the provision of expert advice and the creation of communities of interest. Given the overnight transition, there was little opportunity to give centralized direction, although we closely monitored the situation through regular, and initially daily, meetings with Deans and Senior Faculty.

Related Work
In 2020, the COVID-19 pandemic globally impacted the educational sector, and even institutions without any history of online teaching had to make urgent decisions to move online for the continuation of academic activities. According to UNESCO (2020), over 1.5 billion learners in 165 countries are affected by COVID closures, some 87% of the world's student population. The sudden and substantial shift from the normal academic teaching-learning process to a distance-learning mechanism has unexpectedly created a new phase of teaching and learning.
Of course, online learning has been adopted by many universities across the world. This is a consequence of rapidly developing ICT, including Personal Computers, Tablet, and Smart Phones, as well as ubiquitous wired and wireless Internet access (Alfahad & Almosa, 2002;Yusuf, 2013). Many universities over the past decade have moved programs online and done away with face-to-face delivery. Examples of this include Tsinghua, Peking University, Harvard, MIT, Yale, Oxford, and Cambridge (Bao, 2020;Picciano, 2017). Filius et al. (2019) noted that significant planning and investments are required to go fully online.
Keengwe and Kidd (2010) emphasize that faculty should not only master the technologies associated with online learning, but also transform their pedagogy to support the instructional needs of online students. Hodges et. al. (2020) argued on the experiences of transformed online learning during the COVID-19 period to those courses that are designed to be offered online. The authors pointed out that the universities/colleges should comprehend those transformations while the remote teaching learning is evaluated (Hodges et. al., 2020).
The authors point out the lack of the pedagogical skills of teachers in some universities and its consequential impact on online teaching during the pandemic situation and the importance of preparing the university teachers with skills to deal with the online teaching (Chrysi et al., 2020;Ocak, 2011;Ching et al., 2018;Bates, 2020),. A survey on the impact of COVID-19 conducted amongst final-year medical students in the UK showed that the majority of students feel less prepared for work as a doctor as a consequence of online learning (Choi et. al., 2020).
The literature suggests there are clear lessons for institutions such as ours that switch from almost entirely face-to-face to online teaching overnight. Staff skills, student acceptance, and different student learning styles require careful monitoring if we are to get this right. Consequently, we developed detailed surveys to ascertain our performance in these areas.

Methodology
Given the nascency of the circumstances and our inexperience in online learning, we believed quantitative data would help make sure we were being effective. Our longer-term goal was to move beyond a fire-fighting mode of operation and towards a more measured assessment of what was working well and what was not. This proved prescient for Semester 1 2020-21 where online learning continues to dominate our provision.
Accordingly, our Directorate of Quality Assurance and Enhancement was charged with developing a detailed set of questions to ensure our students were succeeding. There were three broad areas we wanted to understand more clearly, namely: • Utilization: By this, we mean the hardware and software infrastructure utilization, including platform, Learning Management System, network requirements, and variation of system load over time and topics. • Interaction: We were delivering both prerecorded and interactive teaching. We wanted to understand their suitability for different topics and the students' preference for the various formats. • Effectiveness: We wanted to make sure our students were continuing on their learning journey.
A survey was initiated after two weeks of implementation of e-learning across the four campuses of the University to analyze the basic acceptance of distance learning methodologies from the learner's/student's point of view. The outcome of the survey initiated necessary actions to provide more interactive and acceptable methodologies of learning to the students. The continuous distance learning methodologies adopted during this pandemic demanded continuous monitoring of the learning-teaching activity across our campuses. A second survey was launched after five weeks of starting e-learning across the campuses.
Since quick actions needed to be taken for improvement, the two sets of surveys were open only for four days. The responses were collected and analyzed by the Quality Assurance Unit. The first survey responses were presented before the Deans of the Colleges. As a result, rapid actions could be taken for improving the effectiveness of teaching.

Results
A total of 1171 students participated in the first survey, which was launched on April 1, 2020, and 580 students participated in the second survey, which was launched on April 21, 2020, from a student population of 3749. Since the spring semester was expected to be completed in May 2020, immediate actions had to be taken for rectification for the benefit of the students. Table 1 provides the dates of the surveys and the response rate from the different colleges/schools of the University.  Table 2 corresponds to the platform/software used by the students for online learning purposes. The purpose-made platforms used in various colleges of the university include Blackboard and Webinar. Table 2 shows that the preferred mode of delivery is WhatsApp, and we believe this is primarily due to student familiarity and the ubiquity of the platform. NU is a merger of three institutions. A consequence of this is the existence of legacy LMS. However, no matter which LMS the constituent colleges deployed, it was clear that students preferred WhatsApp as their platform. This is certainly related to the use of mobile platforms, but it is not clear why. It could be (a) a consequence of the preference for a mobile platform, (b) the familiar application creating the preference for a mobile platform, or (c) the tight integration of application and handset (cellphone) providing a convenient platform for our students. Whatever the reason, WhatsApp generated effective engagement.
The alternative platforms used were Blackboard, which is used by our engineering college, and Google Classroom, which is used by our foundation college. SOLE is a custom LMS from our Medical College Partner, West Virginia University. The switch to interactive and synchronous learning can be seen in the significant increase in the use of Webinar and Zoom over the three weeks, although it is important to note this has not come at the expense of WhatsApp engagement. Figure 1 shows the students' assessment of the speed of internet connectivity available to them away from the University. At the start, the majority of students considered internet speeds to be poor or very poor. This was mitigated over time, but even so, nearly a third of all our students found internet speed to interfere with their learning. Oman has a developed digital communications infrastructure. Fixed-line connectivity and fibre are available in most conurbations, and the 4G digital mobile network covers 95% of the population and indeed has the fastest mobile data network in the GCC region (Ooredoo Oman, 2019;Ooredoo Oman, 2018). Furthermore, access to educational sites (.edu.om) was free as it did not come out of the individual's data allowance.
Our initial position was that connectivity would likely only be a problem for a small number of students and we would handle that on a case-by-case basis. In reality, around one in five of our students stated they had difficulty accessing online material. Furthermore, as can be seen from Figure  1, initially (Phase 1) over half of our students considered access to be too slow. There is some dissonance between student experience and network capacity. Average download speeds were

Internet Connectivity Speed
Phase 1 weighted avg Phase 2 weighted avg reported by Ooredoo, a local telco, to be around 14Mbit/S, which is sufficient to receive in real-time a MPEG encoded video stream, where typically 2Mbit/S suffices and would be the most bandwidthdemanding requirement we make.
We do not think this is entirely (or even primarily) gamesmanship. There were significant and sustained concerns from students across all campuses that they were simply unable to access the content we were producing. Limited connectivity was a substantial and unexpected blocker at the start. Things improved when we lent out 4G dongles to allow students to use with their laptops. The ingenuity of our students came into play, and the support of the community (rural malls would find a quiet place for students to use the WiFi there) helped people access the material. As a result, the proportion of students reporting slow bandwidth dropped from over half to around one third. This started to work a little better for material that can be downloaded once and then shared via Bluetooth or dongle. However, the subsequent shift in demand to interactive teaching again placed pressure on network availability and speed for a substantial minority of our students, and this remains an issue today. Figure 2 shows the students' assessment of the quality of teaching materials received online. We had substantial issues at the initial stages with over half of our students considering the quality of the teaching to be poor or very poor. The investigation into this showed two distinct concerns. This prompted a series of focus group activities to determine the causes of this.

Figure 2. Quality of teaching materials received online.
Firstly, we had assumed that our students would prefer pre-recorded material that they could access at a convenient time and repeatedly. This was not well-received by students, not just because of unfamiliarity, but also because of a feeling they were being unsupported in their learning journey. Secondly, there were concerns about the technical quality of the material. While we converted 10 classrooms to allow faculty to record material in a professional setting, concerns about contagion, supporting their family, and to some extent inertia meant that much of the material was recorded via a laptop at home. Video quality and especially audio quality was a matter of concern for our students. Restorative action brought these figures down to around 25%, but there still remains a worryingly high proportion of students who struggle with the learning material provided. Figure 3 shows that access via mobile devices remains the most popular form factor for our students. At the start, two thirds of our students accessed material via a mobile device. Tablets were used by a smaller proportion of students (less than one fifth), and laptops were used by around one half of students. Evidence suggests that most PC access was from our medical school where students are supplied with a laptop that has all learning materials for the year on it. Our engineering college saw much less use of laptops. In some households, there was no PC available or it had to be shared amongst other household members. Note that it is not unusual for multi-generational family members in Oman to live in the same property with as many as 20-25 residents. For some of our students, mobile access was out of preference, and for others, it was out of necessity. Either way, this media consumption is in contrast to our content creation processes, which almost always presumes a PC. We suspect we are not alone in such a presumption. Over time, there was a shift to the increasing use of PC's, but the use of mobile devices certainly dominated the early stages. Over the coming academic year, our Teaching and Learning Directorate will drive a mobile-first initiative determining how best to configure our online learning resources for mobile consumption.     Figure 2 that there were concerns about both the clarity and the organization of the material. For instance, fewer than 30% of our students in the initial stages thought the material we were delivering was clear and well-organized. Recording lectures on laptops simply does not work. Expecting young people to watch four or five 50-minute lectures per day in a second language is unachievable.
The remediation we took here was two-fold. Firstly, there was a switch from pre-recorded material to interactive teaching (often a faculty member voicing over PowerPoint with plenty of pauses for questions allied to extensive use of self-organized WhatsApp groups for discussions), which helped deliver the material in a more comprehensible manner. Secondly, the provision of high-quality microphones to faculty to permit better voiceovers was perhaps the single most effective investment we made. Figure 5 provides some unexpected observations. The majority of our students reported difficulty in completing the study tasks given as part of the online lecture. We do not know precisely why, but it is a pervasive phenomenon. At the start, fewer than 20% of our students could complete the assigned out-of-classroom work. The surprising fact is that it was precisely the same work we would have assigned in a face-to-face format. This was not a consequence of ability, either; even our stronger students were giving us this feedback. They said it was too much to do, they were too tired, and the task was too difficult.

Figure 5. Completion of tasks at the end of a session.
In the absence of any identifiable reason for this, our approach was simply to reduce the coursework given out, typically by 30-50%, although it varied greatly by subject since some faculty stopped assigning it altogether. While this resulted in some improvement, we have concluded that the scope and scale of the coursework given to our students need to be markedly different. We intend to examine this matter in more detail during the first semester of the academic year 2020-21 to see if we can identify underpinning causes. Figure 6 demonstrates unequivocally that our students prefer interactive learning. At the start (Phase 1), fewer than half of contact hours were interactive. We had significant pushback from our students that they did not appreciate recorded material. Two weeks later, four out of every five sessions were interactive. Web-based video conferencing and WhatsApp chat groups dominated our delivery mechanism.    Figure 7. Overall effectiveness of online teaching-learning (1=Very Poor, 2=Poor, 3=Adequate, 4=Good, 5=Very Good).
Over half of our students at the start considered our effectiveness to be poor or very poor. After three weeks, four out of every five students were finding our teaching to be of a satisfactory or better standard. This improvement occurred as a result of us being able to monitor our progress closely and through targeted changes. Principal among these were (a) the move to interactive sessions, (b) supporting home-based faculty with better technology, and (c) recognizing quite how hard our students found online learning and adjusting the volume of material accordingly.

Analysis
In the early stages of online teaching, we had issues with the quality of our materials. This alarming fact was likely due to a combination of the following reasons: • The immediacy of our transition to a 100% online model created a substantial amount of unexpected demands on our faculty. • Our faculty were typically inexperienced in online teaching, and this combined with the lack of preparation time impacted quality. • Internet connectivity issues experienced by students, and in some cases by faculty, impinged on construction and delivery. • Access to professional quality AV equipment was limited at first. Most laptops are not designed for broadcast-quality video and audio.
After three weeks, however, we had made significant improvement in the quality of our material through the following actions: • Our IT department augmented our server and bandwidth capacity and installed superior quality audio and video recording facilities in 10 classrooms for our faculty to use. • We acquired additional AV equipment and loaned out to faculty who either preferred to or were required to work from home. • Our faculty formed communities of interest where experienced faculty provided support, advice, and encouragement to other faculty, which helped raise quality significantly.
It may be of interest to note that the most widespread complaint was of poor audio quality. Consumer-grade laptops do not prioritize audio quality, and domestic settings are sometimes noisy or acoustically bright. The provision of broadcast-quality microphones was the single most effective change we made in improving the quality of the materials we produced.
Our students found interactive sessions to be more informative than pre-recorded ones. As a result, our faculty altered their teaching to focus more on interactive sessions. A useful lesson learned here was that our investment in high-quality classroom recording was not as utilized as initial thoughts suggested. We wasted time in doing this when perhaps improving network bandwidth and homebased platforms would have been more impactful. Having said that, our experience over time was that while interactive was preferred, using high-quality audio recording over a PowerPoint presentation was appreciated by our students. It seems that the audio quality had a decisive role in making this format more accepted by students. This seems underreported in the literature.
A frequent concern of students was of being overworked with too many assignments and with too little time to complete them. This was unexpected as we had altered neither assignments nor deadlines. These concerns were consistent across the ability range of our students, subject, and level. It was an authentic and widespread phenomenon. We do not have evidence as to the cause of this, but possible reasons include (a) unfamiliarity with or ineffectiveness of our online learning material or that (b) studying from home was less conducive to learning. A typical Omani family is significantly larger than families in the USA or Europe, with an average of five children per household.

Concluding Remarks
The rapid transition to online learning certainly tested our IT and pedagogical capabilities. It ran hot at times but was rapidly able to adjust and deliver the needed instructional experience. In doing so, it challenged a number of our assumptions. Mobile access is far preferred by our students to PC, yet we generate content for the latter. Commercial delivery platforms, such as WhatsApp and to a lesser extent Zoom, are more readily accepted by our students than the existing LMS, yet we require our students to engage with the latter. Synchronous instruction engaging students more than asynchronous suggests that the traditional face-to-face instruction will not fade away. The provision of highproduction quality asynchronous material does meet with approval, however.
Finally, students seem to learn at a slower pace (and perhaps in a different way) using online instruction. Whether that is a persistent circumstance or an artefact of novelty, is certainly worthy of further research.