Dóra HEGYESI & Ágota KRISZTIÁN

Gamification: is it suitable for all students?

Introduction

In the first school year after an extended COVID-19 lockdown, motivating children and making them collaborate after months of isolation was challenging. Moreover, in Hungarian vocational education and training, students can only learn the science subjects related to their vocation next to mathematics and a subject called "complex science," which is supposed to teach the basics of all STEM subjects combined. In this system, students have very little chance to understand complex processes, and this often leads to a high level of anxiety connected to these subjects.

Due to this situation, vocational students' motivation in STEM subjects - even if they are related to their future profession - could be more robust. This demotivation led to many students failing Geography in the examined vocational school, to which gamification was considered a viable solution. It led to two hypotheses. Firstly, a gamified learning environment will significantly affect the learners' motivation (H1). Secondly, students with different learning orientations or motivational styles will interact significantly differently with gamified learning (H2).

Literature Review

Motivation is one of the most popular interest fields in research of the past few decades for two reasons: it not only affects cognitive and psychological processes, but in today's consumer society, it is an essential trait as it is related to and determines one's feeling of self-efficacy (Ryan & Deci, 2000). A person's motivations are mainly formed throughout childhood and remain approximately the same throughout one's life (Atkinson, J.W. 1988). Motivation has two types: extrinsic and intrinsic. People experience intrinsic motivation when they perform an activity out of interest or enjoyment. Extrinsic motivation is a rather heterogeneous term that refers to any motivation that does not solely stem from inherent satisfaction, e.g., when one's main motive is to avoid failure (Ryan & Deci, 2000).

Kozéki and Entwistle (1983) state that affective, cognitive, and moral motives significantly affect students' in-school behavior. Their motivation is affected by five dimensions: a positive attitude towards school learning, the motive to avoid failure, curiosity, conformity, and the need for social recognition. Three types of students can be identified through these motivational dimensions: followers, performers, and enquirers (Kozéki & Entwistle, 1984). Follower-type students prefer detailed instructions and try to avoid individual work and failure. Enquirer students usually only participate in projects connecting to their interests, but if they do, they enjoy taking the lead. Performer-type students are mainly led by their potential success, but they tend to take responsibility for their learning so that a gamified system might help them the most. They also identified a subscale that shows whether a person experiences high outside performance pressure.

As a supplement to the motivational types, three learning orientations were also identified to gain more information about the traits and attributes of the examined learners: reproductive, organized, and deep learners. Reproductive learners' main goal is fulfilling the teacher's requirements, thus avoiding failure, but in extreme cases, these students are prone to high levels of anxiety. Students with an organized learning orientation seek a system in their studies; they find interdisciplinary connections easily and manage their time well. However, they mainly learn for rewards. Learners with deep orientation enjoy challenges and exploring on their own, though they tend to learn only those subjects that they specifically find interesting. A separate subscale identifies the level of instrumentality (how much one thinks of acquired knowledge as a tool to obtain something) regarding one's motivation (Kozéki & Entwistle, 1984). These two tools combined are regularly used in Hungarian educational research, for example, as a diagnostic tool to develop better differentiation methods in language learning (Péter-Szarka, 2010) or to explore the underlying motivational components of students doing either recreational or competitive sports (Szűcs, 2016).

The connection between motivation and gamification

Based on the Self-determination Theory (Ryan & Deci, 2017), the need to develop ourselves is an innate trait. Thus, students who experience autonomy and self-efficacy during their learning process develop higher and predominantly intrinsic motivation.

Experiencing a classroom climate that supports autonomy and self-regulation (such as a gamified context) often results in higher motivation and better grades, even in STEM subjects (León et al., 2015). Since STEM courses often arouse anxiety in students, it is essential to introduce a system where motivation and autonomy can increase because anxiety and demotivation can lead to low performance. Furthermore, low participation during the lesson might negatively interfere with mastering key concepts (Ababio, 2013). Gamification is the implementation of game elements in a non-game environment (Al Fatta et al., 2019). It is now a widespread method in education however the usefulness of gamification is controversial.

According to Borys and Laskowsky (2013), who compared students' performance through traditional courses and gamified ones, gamification increases the involvement of students throughout the semester. However, it does not affect their academic performance. In theory, gamification is ideal for raising motivation levels since the critical elements of motivation are emotions and interest. In contrast, previous meta-analyses support the cognitive benefits of gamification (Karakoç et al., 2020; Wouters et al., 2013) but not in the area of motivation (Wouters et al., 2013).

From a cognitive point of view, gamification's clear guidelines and goals enhance motivation. Behaviouristic theories suggest that gamification's instant gratification system also supports the growth of motivation levels (Sailer et al., 2015). However, suppose students get points or rewards for every activity. In that case, it might push their motivation towards being extrinsic, resulting in students working for an external reward rather than for developing themselves.

Thus, gamification can quickly become extrinsically motivating and only sometimes lead to student learning (Hung, 2017; Mayer, 2020). Some possibilities for avoiding this are emphasizing collaborative methods or escape rooms, in which students have to compete against the game or the time limit and not against each other (Hanus & Fox, 2015). This is a more progress-oriented approach that highlights collaboration rather than instant gratification.

In a university setting, gamification can raise motivation for a particular subject. Moreover, it helps students develop predominantly intrinsic motivation towards that subject due to their experienced self-efficacy, rooted in the small successes obtained during the semester (Banfield & Wilkerson, 2014). A teacher has many instructional and methodological possibilities to enhance intrinsic motivation, such as structuring lessons consistently or introducing experiments and inquiry-based learning into the classroom (Komsoon, 1999).

Even though many research papers focus on comparing the motivational effects of traditional and gamified learning, the problem of learner types is only sometimes included in these papers, even though this factor extensively interacts with one's motivation. Regarding young adults (university students), there are recommendations for identifying learner types and player types since tailoring the game elements and the course structure around their specific needs might increase their learning efficacy (Abdollahzade & Jafari, 2018). In addition, most experiments are performed in a university setting, where students are generally motivated about their studies. At the same time, this research was conducted in a vocational school, where attendance is compulsory for the students, no matter their interests.

Methods

Research questions and hypotheses

The research was built around the following research questions and hypotheses:

• Q1: Do students with different motivational styles or learning orientations react differently to a gamified classroom context?

• Q2: Are there any motivational components that might shift towards the negative due to a gamified learning environment?

• H1: Students' motivational style and learning orientation affect the motivational response to gamified learning.

• H2: Students with deep learning orientation will experience significant positive change in their intrinsic motivation.

• H3: Not all learning orientations will react positively to gamification.

Sample Size and Sample Selection

The experiment was conducted in a vocational school of tourism and hospitality in the capital city of Hungary, Budapest. Two unrelated classes were picked for the research: a 10th-grade class on the tourism track and an 11th-grade class on the hospitality track. Students of both classes had two geography lessons per week. The classes were chosen by convenience; however, these two classes were chosen from all the possible participants because this combination provided the most significant possible heterogeneity to the sample. All students participated on their own, as well as with parental consent. The research ethics board allowed the study since the course's learning outcomes were still provided, and students could choose whether to continue learning in a gamified context or return to the more traditional evaluation methods.

The research participants were the students of the two classes whose parents allowed the data collection and who could participate in both of the data collection sessions. Twenty-one 10th-grade and eighteen 11th-grade students participated throughout the process (n=39) due to a COVID-19 wave during the research period.

Research Design

The research was conducted from September 2021 to January 2022. During the examined period, both classes covered two units of the same complexity in geography. The first unit material was traditionally presented (frontal, pair, and group work), and students received marks for each of their tests and projects, which were all compulsory. At the end of this unit, students' learning orientation and motivation style were identified using the tests developed by Kozéki and Entwistle (1984), and they filled out two questionnaires to monitor their motivation levels. The first one, the Instructional Material Motivation Survey (Keller, 1987) (from now on IMMS), assesses the factors in an instructional process that lead to increased motivation: attention, confidence, relevance, and satisfaction. The second one was a revised version of Fachraini's (2017) test on motivation type, assessing whether the participant experiences intrinsic or extrinsic motivation dominantly. The revision narrowed down the scope of the test to geography learning instead of general education-related statements.

The second unit was conducted through a gamified context. Students had compulsory and facultative tasks, which they could even design. For the tests and assignments, the learners received points, which they could exchange for certain benefits during the lessons, and the sum of their points provided their grade for the unit - so that the system could be transferred into the traditional educational requirements. The amount of points gained to reach a particular grade was known to the students from the beginning of the semester. Game elements were introduced during the lessons, such as board games, escape rooms, virtual field trips, and trivia games. At the end of this second unit, IMMS and motivation type were reassessed, and students could vote on which instructional method they would prefer during the next semester. The two motivation level tests were chosen to provide considerable heterogeneity to the dataset.

Data analysis

The data was analyzed using JASP. Saphiro-Wilk tests were performed to explore the data's distribution. After the assumption of a normal distribution, paired samples t-tests were performed to explore significant changes between the beginning and the end of the examined learning period. Based on these results, one-way ANOVAs were performed to further investigate the shift of motivational components separately.

Results

Descriptive statistics

Of the 39 participants, 20 belonged to the deep learning orientation, eight fell into the organized category, and 11 belonged to the reproductive orientation. It suggested a very interesting experiment outcome since, theoretically, students with deep learning orientation especially require autonomy and challenges throughout their learning process, which were provided through the gamified system. However, those students who have a reproductive learning orientation might experience a loss of motivation in a gamified context because of the more significant responsibility. The instrumentality factor, which shows whether students are primarily motivated by external rewards, had a mode equal to or higher than three in 12 sample cases. The two examined classes were very different in this aspect. While the majority of the 11th graders had deep learning orientation (14 out of the 18 participants), in the 10th-grade class 9 students belonged to the reproductive category, and six-six students to the deep and Fig. 1. - The composition of the 10th-grade class based on learning orientation Fig. 2. - The composition of the 11th grade class based on learning orientation organised categories.

Fig. 1. The composition of the 10th-grade class based on learning orientation

Fig. 2. The composition of the 11th grade class based on learning orientation

Regarding motivation type, the dataset was rather polarised. Twenty-three of the participants belonged to the performer type, 14 to followers, and 2 to inquire. Performer types usually take responsibility for their learning and prefer to plan, so an increase in their motivation was expected. Interestingly, the "pressure" factor, which measured the extent to which students experienced environmental stress regarding their studies, had an equalling to or higher mode than 3 in 14 cases at the beginning of the experiment, out of which 13 were performer students. These students' feelings of self-efficacy will grow during the examined period. The students who belonged to the inquiring type were not analyzed separately due to the small size of the sample.

Since all research tools in this study include five or more items per category, the variables can be treated as continuous, despite Likert-scale being traditionally treated as ordinal (Sullivan & Artino 2013). For a possible paired samples t-test a test of normalcy was done on the difference of variables between the first and second assessments. The Saphiro-Wilk tests' null hypotheses with a confidence interval of 5% can be rejected, suggesting the normal distribution of the dataset (Table 1). This allowed the performance of paired sample t-tests.

Table 1. Descriptive statistics of IMMS and motivation type subscales (change between start and end of experiment) with a confidence interval of 95%

Results of the paired sample T-tests

Table 2. Results of paired sample t-tests divided by learning orientation

The paired sample t-tests brought varying results. Regarding learning orientation, unexpected results can be seen. Students with deep learning orientation experienced statistically significant change indicating a shift in the positive direction on all measured subscales except for extrinsic motivation. Per contra, organised students showed significant change in extrinsic motivation but not on the intrinsic subscale, despite the fact that all their IMMS components increased significantly except for attention. Even more interestingly, reproductive learners experienced no significant change on any of the subscales, but there is a possible decrease of confidence in the case of these learners based on their needs mentioned in the literature review (Kozéki & Entwistle, 1984). This latter result led to a more in-depth analysis of the results through one-way ANOVAs to further investigate the differences in the changes between the students with different learning orientations or motivational styles during the studied learning period.

Table 3. Results of paired sample t-tests divided by motivational style

Regarding motivational style, both performer and follower students showed significant changes, however, in different subscales. In the case of performer students, a significant positive change can be proved on the subscales of intrinsic motivation, attention, satisfaction and relevance. Follower students showed so significant change in intrinsic or extrinsic motivation, but they also showed significant positive change on the satisfaction and relevance subscales, and interestingly, their confidence increased significantly during the studied period.

Results of one-way ANOVAs on the changes of motivation levels

The difference of motivation levels (change of averages per person) was calculated regarding each subscale in the IMMS and the motivation type survey, which were the calculated differences of values between the beginning and the end of the examined period. The relevant descriptive statistics can be seen in Table 1.

Since these variables are typically distributed, one-way ANOVAs were conducted, examining each subcategory individually (p values are shown in Table 4).

Table 4. The p and F values of ANOVA tests on motivation type and IMMS results

While the ANOVA showed no significant difference in the change of attention and feeling of relevance between the students with different learning orientations, it can be seen that regarding satisfaction and confidence, reproducing learners experienced change during the examined period (Fig. 3., Fig. 4., Table 5.). Interestingly, during the gamified period these learners' confidence significantly dwindled, but the satisfaction factor of their motivation significantly grew.

Table 5. Results of Tukey Test on Satisfaction

Table 6 suggests that there is a significant difference in confidence between students with reproductive and deep learning orientations (applying a 95% confidence interval). The effect size was collated as part of the Tukey Test for post hoc comparisons. As Fig. 3 shows, reproductive learners' confidence decreased during the examined period, separating this group from the two others regarding this aspect.

When the samples were filtered based on the level of pressure and level of instrumentality, the formerly mentioned phenomena were examined in more detail. When only those students' results were examined who either experienced high outside pressure or high instrumentality (mode > 3), there were no significant differences between the learning orientations. However, when only those students were included in the dataset who did not show instrumentality (p=0.861, variance=0.633), it appeared that the reproductive learners experienced a significant drop in confidence compared to deep learners (ptukey= 0.046, Cohen's d = -1.304). In the case when only students who did not experience high levels of outside pressure were included (p=0.912, variance=0.500), the negative change in confidence of reproductive learners was significantly different both from deep learners (ptukey= 0.020, Cohen's d = -1.667) and organized learners (ptukey= 0.046, Cohen's d = -1.601). When the two filters were applied simultaneously, and only those students' samples were examined, who experienced neither high instrumentality nor high outside pressure, there were no significant differences, but that could be a result of the small sample size in this case (n=15). The other subscales of IMMS showed no difference based on these filters.

F ig. 5. Change of intrinsic motivation grouped by learning orientation

Based on the Saphiro-Wilk test, the normal distribution of the motivation type results can be assumed (p=0.797 and p=0.111), and Levene's Test (p=0.480) also allowed the performing of a one-way ANOVA. During the examination of the motivation type results, a connection could only be found between learning orientation and the growth of intrinsic motivation. The Tukey test showed that only those students' intrinsic motivation grew significantly, who belonged to the deep learning orientation (ptukey=0.032, Cohen's d= -1.054) (Fig. 5).

F ig. 6. Change of intrinsic motivation when only the students with low instrumentality but high pressure included

Interestingly, when the data was filtered, and only the students with a high level of instrumentality (mode>3) were examined, the test showed no significant change in motivation levels. However, when the results were filtered both by the level of pressure and instrumentality, the results suggested that students with a high level of pressure but low level of instrumentality react very differently to motivation, based on learning orientation: such reproductive learner's intrinsic motivation seems to dwindle slightly, while students with profound learning orientation showed an increase in their intrinsic motivation (fig. 6). Unfortunately, the difference between them did not prove to be significant. Moreover, it turned out that no students with organized learning orientation experienced a high level of pressure in this sample.

At the end of the examined period, when the points had to be transformed into grades, a fascinating phenomenon occurred. While both classes voted for the gamified system against the traditional learning environment, not all students could turn this system to their advantage. It is impossible to connect the grades to learning orientations because of the anonymity of the project, but while in the class where most students had reproductive learning orientation, the class's average grade increased by almost 20%, in the class where nearly all students belonged to the deep learning orientation the class's average dropped by 17% during the gamified period.

Discussion

The results may be confusing, considering some contrasting tendencies. Even though most of the secondary literature connects gamification to motivation change, based on this data, a gamified classroom context affects students differently based on one's learning orientation. It is essential to mention that most of the students had the same motivational style, so significant differences between these measures were impossible to prove during this study.

The t-test results show the possibility of improvement in extrinsic and intrinsic motivation for most learners. Interestingly, regarding motivation style, performer students showed a significant increase in intrinsic motivation. It might be connected to the flexible nature of gamification, during which they can make up for worse grades and pick a project to work on based on their preferences or interests. This might be why performer students also showed a significant increase in attention while followers showed growth in confidence. Based on motivational style, nearly all students experienced significant growth on the satisfaction and relevance subscales.

However, if the same group of students is examined from the learning orientation aspect, a more complex situation unfolds. Learners belonging to the deep learning orientation experienced a significant change towards the positive in nearly all components of motivation, except for the extrinsic variable. Organized learners went through positive motivational changes in the extrinsic domain, but also regarding their feeling of confidence, satisfaction, and relevance. However, It is fascinating that reproductive learners were seemingly unaffected by the gamified learning context. Unfortunately, a slight decrease in motivation could be detected (with a higher confidence interval), which suggests that with a bigger sample size, gamification might cause loss of motivation for specific learners.

If one takes a closer look at the needs of students with different learning orientations, that might lead to an explanation. Students with a reproductive learning orientation usually try to avoid failure with hard work and much supplementary practice. In the case of the 10th graders, in which class there were many students with this orientation, several students submitted all of the assignments - even though not all of them were compulsory, thus collecting many points, which led to better grades. It could have raised their level of satisfaction, since their hard work brought a result. However, it has to be highlighted that while the change inin satisfaction brought significant results with the students having deep and organized learning orientations, based on Fig. 4, it is suspected that on a bigger sample, a significant decrease in satisfaction could be proven regarding all the other learning orientations.

On the other hand, in a gamified context, the reward comes in the form of points, and students expressed their insecurity about this at the end of the examined period. It was problematic for them to communicate gaining points towards their parents, who were used to getting grades for each assignment or test. However, more is needed to fully explain the phenomena since the significant drop appeared in the case of those students who experience no instrumentality or no outside pressure. Achievement-oriented students appreciate or accept the point system more than those students who are not that focused on performance. Since gamification is performance-focused, it could be a demotivating environment, especially for those students who do not prefer learning individually with less guidance. Interestingly, Entwistle et al. (1987) have found a group of students with reproducing learning orientation who experienced high anxiety levels and outside pressure regarding their studies despite having better than average academic results during their research, which could also connect to the phenomena above regarding the decrease confidence, despite the lack of high anxiety levels in this case.

In the case of the motivational type, it is to no surprise, that students with profound learning orientation, who enjoy challenges and exploration, experienced a more significant increase in their intrinsic motivation. HoweverThe increase of motivation may not be enough for better academic results. In the case of the 11th-grade class, consisting of students with profound learning orientation mainly, participation during the lessons increased. However, even the growing intrinsic motivation was not enough to make the students hand in the assignments, which led to a significant drop in their grades. Based on the in-class discussion, they enjoyed the lessons and the methods of gamification, but this only raised their interest in geography, not in getting better grades.

Conclusion

In conclusion, even though this sample cannot be considered representative, it might suggest that gamification is only equally beneficial to some students. Learning orientation might affect the changes in motivation when one is learning in a gamified classroom context. While some learners might experience an increase in their intrinsic motivation, others might lose their confidence in themselves. Interestingly, while Entwistle et al. (1987) found that those reproducing learners' confidence drops who experience a high level of outside pressure induced anxiety, the opposite results appeared in this research. Nonetheless, this might be a very problematic aspect of gamification since if it makes one lose their feeling of self-efficacy, in the long term it could result in instrumental motivation - as autonomy and the feeling of self-efficacy are necessary for intrinsic motivation, or the loss of motivation all together.

Moreover, the student's academic outcomes should be taken into account as well. Based on these results, students with a reproductive learning orientation could grasp the opportunities and flexibility provided by gamification, which led to better grades. On the other hand, those students who belonged to the deep learning orientation didn't tend to utilize these possibilities.

The feeling of pressure was only assessed at the beginning of this research, but in later cases, it might be interesting to reassess that factor during the second data collection session. Based on the aforementioned results, different tendencies could be expected for specific learning orientations. To sum up, this study shows that gamification has not only benefits but also possible risks.

As a limitation of the research, the sample size was not representative to the school or to a general population of students, only regarding these two examined classes. On a bigger sample size, more refined connections could be discovered. The inner connections of the examined subscales could be examined through exploratory factor analysis in the long run and with a bigger sample size, identifying precisely what element of gamification leads to an increase or a decrease in motivation and which circumstance led to a slight decrease in this study.

References

• Ababio, B. T. (2013). Motivation and Classroom Teaching in Geography, International Journal for Innovation Education and Research Vol. 1-03, pp. 26-36.

• Abdollahzade, Z. & Jafari, S. M. (2018). Investigating the Relationship between Player Types and Learning Styles in Gamification Design. Iranian Journal of Management Studies (IJMS), 11(3), 573-600. https://doi.org/10.22059/ijms.2018.256394.673107

• Al Fatta, H. & Maksom, Z. & Zakaria, M. (2019). Game-based Learning and Gamification: Searching for Definitions. International Journal of Simulation: Systems, Science &Technology. 19, https://doi.org/10.5013/ijssst.a.19.06.

• Atkinson, J. W. (1988). A kockázatvállaló viselkedés motivációs meghatározói. In: Barkóczi Ilona és Séra László (szerk.): Az emberi motiváció I-II. - Tankönyvkiadó, Budapest. pp. 179-201.

• Banfield, J. & Wilkerson, B. (2014). Increasing student intrinsic motivation and self-efficacy through gamification pedagogy. Contemporary Issues in Education Research (Online), 7(4), p. 291.

• Borys, M. & Laskowski, M. (2013). Active Citizenship by Knowledge Management and Innovation. In Implementing game elements into didactic process: a case study - Zadar, Croatia. pp. 819-824.

• Entwistle, N., Kozéki, B., Pollitt, A. (1987). Measuringstyles of learning and motivation. European Journal of Psychology of Education, 2(2), 183-203. https://doi.org/10.1007/bf03172647

• Fachraini S. (2017). An Analysis of Students' Motivation in Studying English (A Survey Study at Uin ArRaniry Banda Aceh). Getsempena English Education Journal, 4, pp. 47-57.

• Hanus, M. D. & Fox, J. (2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers & Education, 80, pp. 52-161.

• Hung, A. C. Y. (2017). A critique and defense of gamification. Journal of Interactive Online Learning, 15(1), 57-72.

• Karakoç, B. & Eryılmaz, K. & Özpolat, E. T. & &Yıldırım, İ. (2020). The effect of game-based learning on student achievement: A meta-analysis study. Technology, Knowledge and Learning, 1-16, 207-222. https://doi.org/10.1007/s10758-020-09471-5

• Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development 10, 2.

• Koomson, A. (1999). "Basic teaching techniques" in Educational Psychology Handout for PGDE Programme (Unpublished): University of Cape Coast.

• Kozeki, B. & Entwistle, N. (1983). Describing and utilizing motivational styles in education. British Journal of Educational Studies, 31(3), 184-197 https://doi.org/10.1080/00071005.1983.9973658

• Kozéki, B. & Entwistle, N. J. (1984). Identifying dimensions of school motivation in Britain and Hungary. British Journal of Educational Psychology, 54, 306-319.

• León, J. & Núñez, J. L. & Liew, J. (2015). Self-determination and STEM education: Effects of autonomy, motivation, and self-regulated learning on High School Math Achievement. Learning and Individual Differences, 43, 156-163. https://doi.org/10.1016/j.lindif.2015.08.017

• Mayer, R. E. (2020). Cognitive Foundations of Game-Based Learning. Handbook of Game-Based Learning. MIT Press.

• Péter-Szarka, S. (2010). Az idegennyelv-tanulási motiváció és az azzal összefüggésben álló tanulói jellegzetességek változása a felső tagozatban. Iskolakultúra, 20(9), 105-117. https://ojs.bibl.u-szeged.hu/index.php/iskolakultura/article/view/21060 (2024.06.04.)

• Ryan, R. M. & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54-67.

• Ryan, R. M. & Deci, E. L. (2017). Self-determination theory: Basic psychological needs in motivation, development, and wellness, 3-19. https://doi.org/10.1521/978.14625/28806

• Sailer, M. & Hense, J. & Mandl, H. & Klevers, M. (2015). Psychological Perspectives on Motivation through Gamification. pp. 1-10.

• Sullivan, G. & Artino Jr. A. R. (2013). Analyzing and Interpreting Data From Likert-Type Scales. Journal of Graduate Medical Education. 5(4), pp. 541-542.

• Szűcs, K. D. (2016). Az élsportoló diákság teljesítménymotivációja. In "A mi tendenciáink…" Szakkollégiumi Tanulmányok (Vol. 4, pp. 53-69). chapter, Hatvani István Szakkollégium.

• Wouters, P. & van Nimwegen, C. & van Oostendorp, H. & van derSpek, E. D. (2013). A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249-265. https://doi.org/10.1037/a0031311.