Usability of iStudyGuides : A confirmatory factor analysis model

With the advances of technology, technological learning tools are becoming more important in enabling learners to study more effectively. The prerequisite for the success of any technological learning tool hinges on its usability or technical ease of use. A tool that is not usable has the undesired effect of disrupting the user’s learning as more time is expended on learning the tool than the contents (Wong, Nguyen, Chang & Jayaratna, 2003). Therefore, the aim of this research is to explore and uncover the dimensions of the usability of the interactive e-study guide known as the iStudyGuide used as one of the main learning resources in the context of SIM University (UniSIM). Therefore, the research question in this study is “What are the dimensions that determine the usability of iStudyGuides?”. It employs a survey research methodology, utilising the statistical methods of exploratory and confirmation factor analysis. Data are collected from 278 students who used iStudyGuides in their learning at UniSIM. In the confirmatory factor analysis process, Bollen-Stine bootstrap p = 0.688. CFI = 0.996, RMSEA = 0.026 and PCLOSE= 0.918. The confirmatory factor analysis shows that the final model of usability of the iStudyGuides that comprises of the dimension of reliability, utility, learnability, accessibility and control has a good fit on the data. The emergence of the model of usability of iStudyGuides sets as the basis that iStudyGuides can be evaluated to ensure quality assurance in terms of usability.


Introduction
The study guide is integral to the students' learning experience at SIM University (UniSIM).In particular, it serves as the students' first point-of-contact with a course, and a tool for managing their learning, along in a progressive journey towards acquiring and discovering knowledge associated with a subject matter.At UniSIM, courses' study guides, known as iStudyGuides, come in the form of the EPUB 3 format 3 .UniSIM adopted the EPUB 3 format as the delivery platform for its study guides because of its flexibility and learning enhancing potential.In terms of flexibility, the reflow-able nature of EPUB 3 format allows learners to access their study guides on all electronic platforms, namely hand phones, tablets and personal computers, thus allowing them to learn anytime anywhere.E-PUB 3 also allows rich media and interactive features to be incorporated into the course content.The use of such features in supporting or presenting the content has potential in enhancing the student learning experience.In short, the iStudyGuide is an interactive e-study guide that can be accessed from multiple delivery platforms.Since the iStudyGuide is a technological learning tool, how users perceive its usability or technical ease of use can affect its learning effectiveness.A tool that is not usable has the undesired effect of disrupting the user's learning as more time is expended on learning the tool than the contents (Wong, Nguyen, Chang & Jayaratna, 2003).Therefore, the aim of this research is to explore and uncover the dimensions of the usability of the iStudyGuide in the context of UniSIM.Therefore, the research question in this study is "What are the dimensions that determine the usability of iStudyGuides?".
While there is an abundance of literature in the domain of usability, these concepts or criteria in the domain of usability were not consistently defined or operationalised across models.Besides, they were usually designed to evaluate websites, software or learning management systems instead of interactive e-study guides.Most of the time, it was also unclear how these concepts or criteria could be translated into a metric to evaluate usability.Thus, these issues made it difficult for researchers or practitioners to choose the most suitable concepts or criteria in the evaluation of usability relevant to their own contexts.Therefore, in this study, the researchers decided to include the five most common concepts or criteria that were mentioned in the literature as the hypothesised model of usability in the context of iStudyGuides.Therefore, it is hypothesised that the usability of iStudyGuides is made up of the dimension of learnability, utility, reliability, accessibility and control.These five dimensions, in other equivalent forms, generally encompassed most of the variables investigated by previous studies in the domain of usability.
Learnability is consistently cited in literature as an important and fundamental attribute of usability (Blandford & Buchanan, 2002;Booth, 1989;Hix & Hartson, 1993;Nielsen, 1993;Schneiderman, 1992;Swett, 2002;).It is one of the five attributes of usability pointed out by Nielsen (1993), in addition to efficiency, memorability, error recovery and satisfaction.Likewise, Booth (1989) proposed usability to consist of four factors: usefulness, effectiveness (ease of use), learnability and attitude (likeability).While many definitions of learnability have been postulated, this concept was defined based on the initial user experience in most studies.A system or a tool that is easy to learn allows initial users to attain a reasonable level of usage proficiency within a short time (Nielsen, 1994).The perceived ease of learning a technological learning tool will allow them to devote more time and attention to learning the course materials instead of spending additional time to learn how to use the tool (Davis, Bagozzi & Warshaw, 1992).
Utility refers to the usefulness and relevance of functions in helping students to learn (Bernérus & Zhang, 2010;Brooke, 1991;Zaharias, 2009).The functions in some e-resources may encompass highlighting, memo, copy text, share, define, online search and within guide search.A literature review conducted by Bernérus and Zhang (2010) revealed the presence of learning and authoring support tools to be an essential factor in usability assessment of technological learning tools.Additionally, Zaharias (2009) found empirical support for learning and support tools as criteria for usability evaluation.Utility is measured by the presence or absence of tools such as notes taking, job aids, glossaries that support both individual and groupbased activities (Bernérus & Zhang, 2010;Brooke, 1991).
Reliability refers to the dependability of the technical functions of the e-resources (Constantine & Lockwood, 1999;Nielsen, 1995;Siritongthaworn, Krairit, Dimmitt & Paul, 2006).Reliability comes about when iStudyGuides follow consistency standards.A system that has consistency standards should allow the learner to experience the user interface, encompassing control, colour, typography and dialogue design, in a uniform manner.A reliable system is also one that takes into consideration error management issues, including error prevention, diagnosis and recovery from errors (Constantine & Lockwood, 1999;Nielsen, 1995).Poor availability of access points, slow network communications and a lack of software application were cited as challenges to using an e-learning tool, undermining the reliability of the system (Siritongthaworn, Krairit, Dimmitt & Paul, 2006).
Accessibility refers to the convenience of using iStudyGuides (Yordanova, 2007;Dee & Allen, 2006;Caldwell et al., 2004).Accessibility is measured by the extent of access on a variety of equipment and platforms such as laptops and handheld devices during working, learning and commuting hours.Bernérus and Zhang (2010) reported the inclusion of accessibility as a criterion of usability in 9 out of 27 empirical studies reviewed.Furthermore, accessibility was shown to be an empirically validated criteria in Zaharias ( 2009)'s study.
Control refers to the amount of control that learners have in personalising their learning experience through the use of iStudyGuides (Oulanov & Pajarillo, 2002).Learner-controlled instruction refers to instructional designs where learners have the ability to make decision regarding some aspects of the path, flow or events of instruction.The emphasis of a learnercontrolled instruction is to give learners the freedom to choose learning activities that suit their own individual preferences and needs (Williams, 1996).Elissavet and Economides (2000) argued that learner control is an important factor in hypermedia learning systems, with a primary role in the design of interactive learning as it gives students the freedom to tailor their learning experience to meet their own individual needs.Learner control is measured by the extent of freedom students have in regulating their own learning by exercising choice and discretion over the sequence, pace and amount of information they can process (Chung & Reigeluth, 1992;Milheim & Martin, 1991).This gives learners the ability to make decisions about what sections to study and the sequence in accessing the interactive material.
Although these studies supported the importance of the dimension of learnability, utility, reliability, accessibility and control in the design of interactive study guides or learning systems to learning, the studies did not conduct any further analysis to confirm that each of these dimensions are inter-related and are part of an empirical model that define the usability.As a result, this study aims to close this gap by proposing a model of usability and confirming it through the use of factor analytic methods to answer the research question "What are the dimensions that determine the usability of iStudyGuides?"

Method
iStudyGuides are mobile interactive study guides.Thus they are considered a form of technological learning tool.Predictive evaluation, heuristic evaluation, naturalistic observation, questionnaires, interviews are some of the more popular methods to evaluate usability of technological learning tools (Sharp, Rogers, & Preece, 2007).In this study, heuristic evaluation was first used to ensure rigour in the content validity of the questionnaire items conceptualised to measure the five dimensions of the hypothesised model of usability.After that, the end-users, the students, were invited to participate in the questionnaire survey.This allowed the users' perspectives about the tool to be captured (Baber, 2002).The data was then analysed through the exploratory factor analysis to validate the questionnaire.It is then followed by another round of questionnaire implementation to evaluate the fit of the hypothesised model of usability in the context of iStudyGuides.
Although there were questionnaires (such as Chiu & Hung, 2009;Demers, Weiss-Lambrou & Ska, 1996;Drummond & Themessl-Huber, 2007;Hashim, Wan Ahmad & Ahmad, 2011;Jordan, 2000;Kirakowski & Corbett, 1993;Lewis, 1995;Lindholm, Keinonen & Kiljander, 2003) that were formulated to measure the usability of technological learning tools, they were usually designed to evaluate websites, software or learning management systems.However, iStudyGuides are interactive e-books.Besides, such questionnaires were mostly not statistically validated.Consequently, it was decided that the questionnaire items used to evaluate the features of iStudyGuides are to be conceptualised with due consideration to the literature review, in the context of an interactive e-book.In this study, it is hypothesised that the usability of iStudyGuides is made up of the dimension of learnability, utility, reliability, accessibility and control.Based on the literature, a set of 3-5 questionnaire items was conceptualised to measure each dimension.This list of 26 items were further refined and confirmed for content validity through a team of three experts in the domain of usability.The questionnaire was cleared by the Institutional Review Board at the university.The final list of 15 items (in Table 1) used in the exploratory factor analysis is shown below.
For the first survey, an invitation to participate in the first survey was sent to 1231 students.A total of 48 students participated in the survey and completed it.They evaluated each item on a Likert scale of 1-"Strongly Disagree", 2-"Disagree", 3-"Slightly disagree", 4-"Neither agree nor disagree", 5-"Slightly Agree", 6-"Agree" and 7-"Strongly Agree".They were also asked to comment about the use of iStudyGuides in terms of their usability in an open-ended response item in the questionnaire.An invitation was sent to a total of 5032 students were using iStudyGuides for the first time, to participate in a second survey near the end of the semester.329 students responded to the survey and completed it.The respondent samples for both surveys were fairly representative of the active UniSIM student population in terms of certain demographic and institutional factors 4 .Although MacCallum, Widaman, Zhang and Hong (1999) reported that there have not been much agreement in the research community about the minimum number of subjects needed for factor analysis, a minimum ratio of number of participants to manifest factors of at least 3.0 should be able to yield a recognisable factor pattern.In this research, the ratio of number of participants to manifest factors in the exploratory and confirmatory factor analysis were about 3.2 and 19.2 respectively.

Analysis
The analysis section reported on the validation of the questionnaire items used to evaluate usability in the context of iStudyGuides and the fit evaluation of the hypothesised model of usability.

Validation of the Questionnaire items
A total of 48 respondents' data was used in the first round analysis.The aim of analysis in this round was to validate the questionnaire items that would be used to evaluate the five dimensions of usability in the use of iStudyGuides.The respondents' data from the open-ended response item that was used to solicit comments about the usability of iStudyGuides were first analysed.There were 41 comments and they were mostly related to the hypothesised dimensions of learnability, utility, reliability, accessibility and control.No significant new dimension related to usability was mentioned in the comments.
Factor analytic function in SPSS Base was used in the statistical analysis.The ratio of subjects to variables in this factor analysis process was 3.20.The extraction method, Principal Axis Factoring and the rotation method of promax were used in the analysis.The Kaiser-Meyer-Olkin Measure of Sampling Adequacy was 0.755 and the Bartlett's Test of Sphericity was significant (χ 2 (66) = 466.121,p < .05).The determination of the correlation matrix was approximately 0.00001582.There was 0 (0%) non-redundant residuals between observed and reproduced correlations with absolute values greater than 0.05.These measures assured that the data set here is adequate for factor analysis.A five-factor solution (in Table 2) that accounted for 79.2% of the total variance (in Table 3) was produced with 3 items removed from the original list of 15 items.Each pattern coefficient of the solution was at least 0.506 and a mean pattern coefficient of at least 0.690 was obtained for each factor.The Cronbach's Alpha Coefficient of the items in each factor was at least 0.774 (in Table 2).The correlation between the factors was not more than 0.695 (in Table 4).Thus, the reliability, convergent validity and discriminant validity of the data were generally assured.Table 5 below shows the final list of the items used that was conceptualised the items used to measure the hypothesised model of usability of iStudyGuides.

Fit Evaluation of the Hypothesised Model of Usability
The second round of questionnaire implementation was conducted with the questionnaire validated from the first round.A total of 329 students responded to the second survey.However, a total of 230 responses were used in the confirmatory factor analysis process due to their items' completeness and validity.The aim of this round of analysis was to evaluate the fit of the hypothesised model of usability of the iStudyGuides.
SPSS AMOS was used in the analysis.Maximum likelihood model test was employed.As multivariate normality was violated in the data, bootstrapping was performed to overcome this limitation.The data collected in this stage were used to confirm the model of usability produced through exploratory factor analysis in the first round of analysis.Testing the null hypothesis that the model is correct, Bollen-Stine bootstrap p = 0.688.CFI = 0.996, RMSEA = 0.026 and PCLOSE=0.918.The standardized regression weight estimates of all manifest variables are statistically significant and thus are representative of their latent variable.The estimates of correlations between the latent variables are also statistically significant.In short, the 5 factor model exhibited a reasonably good fit of the data.Below is the diagrammatic representation of the factors.In summary, the hypothesised model of usability of the iStudyGuides that consisted of the dimension of reliability, utility, learnability, accessibility and control, fit the data.

Limitations
Although this study managed to confirm the various dimensions of usability that influence that the use of iStudyGuides, there are several limitations worth noting.First, the study relied solely on self-reported questionnaire survey in the collection of data.This might cause some reliability issues as the respondents might interpret some of the items differently.Thus, further reliability tests can be conducted with the possible complement of other data such as observation of the participants' usage of the iStudyGuides or more structured interviews if resources are not a concern in future.Second, the current results are most probably applicable in the context of iStudyGuides.There is a need to cross validate the findings to that of other interactive e-books.Third, as the five dimensions of usability were suggested by the researchers in this study based on the literature review, there might be a possibility that one or more element that impact on the usability of iStudyGuides was not captured.Therefore, it is important that new dimensions of usability are investigated if they are brought up by stakeholders in future.Fourth, the sample used in this research was made up of non-traditional undergraduates who studied part-time and had an average age of 28 years.Thus, the results might not be representative of traditional undergraduates though it could have some implications on non-traditional undergraduates in other universities.

Discussion
The confirmatory factor analysis has shown that the final model of usability of the iStudyGuides had a good fit on the data.This means that that the latent constructs are good representative of the usability of iStudyGuides.Through the rigorous process of factor analysis, the 12-item questionnaire is considered a valid instrument to determine the level of usability that iStudyGuide offers to the end users.Thus, in the case of this research, the validated model and its questionnaire serve as a robust basis for the quality assurance of iStudyGuides in terms of usability.This is an important process as a tool that is not usable has the undesired effect of disrupting the user's learning as more time is expended on learning the tool than the contents (Wong, Nguyen, Chang & Jayaratna, 2003).At the same time, the validated model of usability of iStudyGuides will enable all the stakeholders, namely the developers, students and instructors in obtaining a common understanding of what constitutes a usable iStudyGuide.
While questionnaires were designed to measure the usability of technological learning tools, they were mostly not specifically designed for interactive e-books or were not statistically validated.Therefore, the model of usability of iStudyGuides serves as the first or one of the few statistically validated models to evaluate the usability of interactive e-books.This validated model of usability can serve as the basis for development of other e-book evaluation platforms or for comparison with other usability model in the domain of e-books.
In conclusion, the study elucidated the dimensions that determine the usability of iStudyGuides.It is hoped that the findings can help to improve the iStudyGuides to enhance the learning experience of the students at UniSIM.

Figure 1 .
Figure 1.Proposed empirical model of usability.

Table 1
List of items under technical domain

Table 2
Five factor solution

Table 3
Total variance explained When factors are correlated, sums of squared loadings cannot be added to obtain a total variance.

Table 4
Factor Correlation Matrix

Table 5
Description of factors in usability