ChatGPT in Computer Science Education: A Case Study on a Database Administration Course

Abstract

GenAI tools like ChatGPT have changed the educational landscape, and empirical experiences are needed to better understand how to use them to their fullest potential. This article empirically explores the usage of ChatGPT 3.5 in database administration education through a case study conducted with 40 computer science students. Specifically, it inspects how widespread the use of ChatGPT is and students’ perceptions of this tool, how prior knowledge on a topic affects the use of ChatGPT, and the relationship between the usage of ChatGPT and success in solving practical problems. The student’s grades in a computer practical exam, a set of theoretical tests to assess progression in knowledge acquisition, and a comprehensive questionnaire are employed as research instruments. The obtained results indicate that students use ChatGPT moderately but more frequently than traditional internet learning resources such as official documentation, Stack Overflow or googling. However, the usage is uneven among students, and those who end up getting better grades use ChatGPT more. Beyond prompting skills, one of the elements that is key to the students’ productive use of this tool is their prior knowledge about database administration. This article concludes that ChatGPT is an excellent educational instrument in the context of database administration and that in order to use it properly, it is necessary for students to have good prompting skills as well as a sound theoretical basis. Training students in the use of GenAI tools like ChatGPT, for example, with a guided practice strategy where prompting and conducted step-by-step practice are employed is key to prevent the appearance of new digital trenches.

1. Introduction

During the last two years, the huge breakthrough of Generative Pretrained Transformers (GPT) on Artificial Intelligence (AI) led to a boost and democratization of the use of Generative AI (GenAI) solutions, in the form of chatbots. Several companies have launched their solutions to the public in order to train, test, and acquire customers for their non-free versions, with Open AI and its ChatGPT, or Google and its Gemini being the most prevalent items. Students around the world are changing their habits, and now these tools are their favorite way to search for information [1]. As a reference, it could be stated that 27% of the students were using AI writing tools in spring 2023, and by that year’s fall season, the percentage had boosted to 49% [2].

Although this is a powerful new tool that can be used for student learning, concerns arise about the possible misuse of received information, such as cheating [3] or taking the received answer without further mental processing [4]. Some actions that can prevent it are being taken into account, such as creating new approaches to academic tasks and assessment [5]. Other efforts are being made to verify the validity of these tools for serving as study partners, based on their performance on exams or practical assignments [6]. Although the role of Generative AI in learning is a controversial topic, due to the risk of undermining student learning skills, there are studies that turn this last point of view upside down, even in memorization skills [7].

Facing this new threat, for some people, opportunity for others, several international higher-education institutions have raised recommendations, following the first studies on the impact, possibilities, and dangers of GenAI use in this environment. One of the most used is the work of Gimpel et al. at the University of Hohenheim [8], where they stressed the importance for students to know and respect regulations, focus on learning objectives rather than on the tool, and appealed to their use of critical thinking whenever they interact with these GenAIs. Meanwhile, for teachers, it is recommended to provide rules from the beginning of the season and to use GenAI to provide personalized training experiences to students, always oriented to improve their critical thinking. Another interesting study is [9], where the authors recommended up to seven possible roles for GenAI when it comes to higher-education classrooms: as a mentor to provide feedback, as a tutor to provide direct instruction, as a coach to give deep reflection on the study and habits, as a team member in teamwork tasks to provide new approaches to the group, as a simulated student to force the real student to explain any item to it and by that means reinforce their learning, as a simulator to act as a practical trainer, and as a tool to improve students (and professor) efficiency. Although there is a full set of example prompts to implement them all, each one of the roles has a counterpart: the narrowing of critical thinking, biases, mismatch with the student, conflicts, hallucinations (unreal answers), low fidelity, or writing-without-thinking danger, respectively.

From then on, the aforementioned higher-education institutions have published recommendations following these lines in several ways. All of them stress the importance of changing some traditional assessments, such as typical written assignments, where GenAI is definitely going to excel, turning them into challenging situations for students in order to appeal their critical thinking [10]. In fact, tracking this student’s thinking process could become a powerful new way of assessment, even with the help of GenAI [11]. Policies for the use of GenAI at all levels are recommended in documents such as [12] in the United States, where a human-in-the-loop strategy is stressed. This is increasingly common in the latest publications and is even regulated by the European Union as mandatory in any kind of assessment that may affect future employability, taking this issue as a red line or reaching a “high-risk” level in the use cases of AI [13]. Other recommendations in [12] are directed towards GenAI developers, to align them with educational purposes from scratch, to implement information and formation policies for faculty members who develop specific guidelines and guardrails, and to enhance trust and safety in the use of GenAI. The responsibility of educational personnel to keep track of these tools and learn their use is depicted in [14], added to department or institution collaborative opportunities to apply them, and promoting cooperation in actions between institutions. All of these works show the importance of counting on students voices and perspectives in every process, as well as reviewing academic programs. Many of the current academic programs will be impacted by the use of these tools since professional activities will include the use of GenAI once students leave universities to enter the labor market [15]. Indeed, this latter study shows not only the risks about the threat that GenAI entails to current jobs that may disappear, or at least be deeply transformed but also the opportunities that GenAI open to more creative, empowering and human-enhancing skilled jobs. Another very interesting recent study shows that students claim that a higher-education transformation is needed for future-ready employment in a GenAI-powered society, with the need for new learning skills and more interdisciplinary and hands-on topics [16]. The trends that students point out are these competences for future workforce, AI literacy and new literacies to come, interdisciplinary teaching, innovative pedagogies, and new assessment formats.

Another major concern is the ethical use of GenAI, including copyright issues, responsible and trustworthy use, privacy and data governance (and not only where the uploaded data end but with what use: further training, publication, information collection, etc.), transparency, accountability, societal well-being, and knowledge of carbon footprint impact of the tools; all this is studied and recommended in the European Union in [17]. There is a real possibility of reopening or deepening the digital gap, which is a priority to ensure equitable access to AI tools on campuses [18]. Even more, these tools can be taken as an opportunity to increase inclusion in higher-education areas by creating specific adapted content or providing GPT assistants for students with disabilities, or removing biases affecting minorities, races, gender, religion, etc. [19].

In addition to these general published recommendations, there has been an increasing number of published papers in 2024 providing insights on implications of Generative AI for education and research, also covering bias and inclusion, ethics and regulation, offering usage approaches as a support system or intelligent tutoring one, or in assessment [20]. For example, ref. [21] is a deep reflection on how universities must creatively revisit their teaching by developing AI-powered learning assistants or adaptive learning systems, and even faculty assistants. It is stated that learning improvements may be added to alleviate the pressure on stressed staff.

More specifically for the present work, when it comes to computer science, these tools are likely going to impact jobs in this area: it is a real concern because not only can GenAI perform programming tasks in a very good way but also many job sectors are increasingly using it [22]. Indeed, AI is reshaping the syllabus of machine learning or data science. Therefore, both the syllabus and the way of teaching computer science are affected and should tend to adapt [23]. The challenge for faculty members is also to stay updated on rapidly advancing technologies, maybe even more critically in this area than in other ones. A balanced approach is needed, integrating AI to enrich learning while carefully managing its implications for educational practices and student development. Thus, misuse can lead to the cheating or narrowing of programming skills of students, but good use can really improve their critical thinking and high-level computer science skills [24].

Recent evidence shows the ability of these tools to solve computer science problems at the higher-education level, as well as the aim of students to adapt them to their study, such as the one performed in the Netherlands in [25]. A good attempt to incorporate GenAI into a renowned online course at Harvard University—the CS50 course—can be found in [26]. Interestingly, they show how the integration of AI can enhance the learning experience, providing students with continuous and personalized support, leaving educators more time to address higher-level pedagogical issues. Additionally, AI tools demonstrate their usefulness in explaining code snippets or improving code style, as well as answering any kind of administrative or curricular queries.

On the other hand, GenAI is a tool that can promote student programming self-efficacy and motivation, provided that teachers take specific actions to teach students prompt engineering skills [27]. It has been shown that students not only need this guide but also demand it when facing the study with the help of these tools [28], showing it to be more necessary, as the programming task to be addressed is more complex and higher level [29].

Diving deeper into the inherent GenAI skills, in [23], there is a description of an experiment using GenAI to solve a computer science complex problem, showing that it is not a good approach, and that it is rather better to divide it into small problems, a skill that students should gain, together with designing programs rather than just programming. At the same time, it shows a reflection about how coding skills can be quickly overcome in the teaching process to focus on understanding processes and the topic foundations. These two conclusions can be reinforced in recent works: Ref. [22] looked for the limitations of ChatGPT in solving software development problems, and it was shown that it is good only at the easy- and medium-level ones. As a matter of fact, a survey shows that programmers are using AI tools to create better-performing, shorter, and more readable code (i.e., to become more productive, not to solve big tasks). On the other hand, Ref. [30] drew similar conclusions about the positive role of GenAI in clearing the learning toward the fundamentals, as well as in solving the calculator dilemma. Relevant for the work presented in this paper, it is applied to the topic of database systems.

Regarding GenAI solutions for creating teaching aids, a very recent paper [31] explored the possibility of using a RAG (Retrieved Augmented Generation) chatbot in a computer science classroom. A total of 46 students participated in this study. The results showed that the chatbot performed with a high accuracy rate (88% of the answers were considered right by teaching assistants based on the course materials), and the students felt more engaged with the more personalized training. However, they also claimed an additional need for human interaction with teachers, who, on their side, certainly reduced their administrative burden. Notwithstanding, once again, critical thinking and the teacher’s guide are mandatory, as the detection of inaccurate statements by a bot-generated tool is not efficient enough. This study not only evaluates the precision of bot responses, but also analyzes student acceptance, congruence with the course material, the comprehensiveness of the support offered, and associated costs, filling some of the gaps identified in the literature on LLM-driven educational chatbots.

Other higher-education institutions are developing some solutions for AI-assisted learning [32]. The European School of Management and Business (EUDE) developed EDU, a virtual co-tutor powered by Generative AI, using IBM’s suite of AI solutions including Watson Assistant and Watson Discovery. This system supports administrative, academic, and logistic queries in real-time, which could be applied to database administration courses. Loyola University Chicago implemented LUie, an AI-powered digital assistant built on the Oracle Digital Assistant platform. Although not specifically for database administration, this system demonstrates how AI can be integrated into university systems to provide continuous student support and personalized information.

Focusing on data management, it has been reported that the GenAI scope can fill the gap when automation fails to understand the semantics of the data. Thus, difficult database problems can take a new approach, boosting data discovery, query synthesis, schema matching, and the like [33]. By their nature, these GenAI tools are predictive models; therefore, they can perform well when it comes to combining them with information retrieval. MIT researchers found that SQL does not allow one to incorporate probabilistic AI models, and, at the same time, solutions with probabilistic models making inferences do not support complex database queries. To fill this gap, they created GenSQL [34], a Generative AI system for databases that allows users to perform complex statistical analyses on tabular data without extensive technical knowledge. This tool could be particularly useful in teaching database administration by allowing students to make predictions, detect anomalies, guess missing values, fix errors, or generate synthetic data.

Ref. [35] explored the use of ChatGPT as an assistant to automatically correct SQL exercises in a university database course. ChatGPT effectively identifies syntactic and semantic errors, provides explanations, and assigns grades with a strong correlation to human teacher ratings, although, interestingly, it tends to be stricter. Its accuracy decreases with more complex exercises due to subjectivity in error assessment. Although ChatGPT excels at detecting syntax errors (88.9%), it often confuses them with semantic ones, and its suggestions are only valuable in about half of the cases. Comparisons of Few-Shot and Zero-Shot Learning reveal that the latter produces clearer explanations, while the former relies heavily on provided examples. The study concludes that ChatGPT shows potential as a teaching assistant, but further research is needed to refine its application in education.

In ref. [36], a chatbot is used to support students in completing SQL queries in a step-by-step process. Students who used it performed better on the final exam (4 out of 10 passed the exam, a number halved among students who did not use the tool), while teachers received insightful performance metrics on student progress, aiding them in instructional decisions. Anderson et al. research also highlighted the critical role of prompt engineering in obtaining successful results with GenAI for SQL practice [37], highlighting that students benefited significantly from teacher guidance in developing effective prompts.

Together, these studies underline Generative AIs potential as an SQL learning aid, while also highlighting the importance of instructor support for effective prompt formulation and student guide for GenAI usage. They also show the need for further research on the utility of these tools in database administration courses, as there are still some open questions about (i) the combined use of traditional and new tools, (ii) the ability of these tools to solve complex problems at the university level, and (iii) the possible improvement in learning database administration using them.

In consideration of the aforementioned findings and previous efforts, we have undertaken an empirical study which addresses the following research questions (RQs):

• RQ1. How widespread is the use of ChatGPT among students to learn database administration, and what is the perceived reliability and usefulness of this educational resource compared to other traditional resources?
• RQ2. How does prior knowledge about database administration affect the use of ChatGPT to solve database administration problems?
• RQ3. Is there a relationship between the usage of ChatGPT and the success in solving practical database administration problems?

This study aims to advance the experience of using GenAIs in teaching computer science, and more specifically database administration, following the aim of the recommendations described above. The rest of the article is structured as follows. The materials and methods are described in Section 2. The results are presented and discussed, respectively, in Section 3 and Section 4. Lastly, Section 5 presents the conclusions, limitations, and future work drawn from this study.

2. Materials and Methods

2.1. Context

This research presents a case study carried out in a database administration (DBA) course during the academic year 2024–2025. The course is part of the fifth semester of the Bachelor’s degree in Technologies for the Information Society delivered by the Faculty of Computer Systems Engineering of the Universidad Politécnica de Madrid (UPM), located in Spain. This mandatory course accounts for 6 European Credit Transfer System credits (ECTS), which is equivalent to 150–180 h of student work. The course deals with traditional database administration topics, such as hardware approaches like RAID to increase database performance and availability, software approaches such as indexes to improve database optimization, or other critical topics such as security, restoration, or monitoring. Although the participating students have previously taken another generic course on databases, where they learned to model a database or perform SQL queries, the topics in the database administration course are completely new to them, so they have no prior knowledge of the subject. During the course, students receive theoretical–practical classes and they also attend hands-on computer classes, in which they receive teacher practical tutorials and work in groups to perform a practical computer assignment. The case study presented here focuses on the utilization and perceived reliability and utility of the resources used by the students to complete the practical computer exam.

2.2. Sample

The sample comprises 40 students enrolled in the DBA course during the academic year 2024–2025, who completed the individual exam. They are 33 males (82.5%) and 7 females (17.5%), and the mean age is 21.78 with a standard deviation of 1.42. Students who do not pass the course the previous year are excluded from this case study to ensure that the students in the sample have no prior knowledge of the subject.

2.3. Procedure and Materials

At the beginning of the course, the students take during an introductory class a knowledge test related to practical work to evaluate their previous knowledge (to view the questions, go to

Table A1. Test carried out throughout the course to measure the evolution of knowledge.

Question/Answers (The Correct Answer Is in Bold)
1. What statement is used in MySQL to load data from an external file? A. LOAD DATA INFILE B. IMPORT DATA INFILE C. INSERT INTO D. Data loading must be done through third-party software, not directly from MySQL
2. What statement is used in MySQL to load data from an external XML file? A. The same statements or tools used to load files of other formats B. LOAD EXTERNAL XML C. LOAD XML INFILE D. To load XML files, the file must be pre-transformed into another format compatible with MySQL
3. Define foreign keys before the loading process A. Makes the loading process faster and more reliable B. Makes the loading process more reliable but slower C. Makes the loading process faster but less reliable D. Makes the loading process less fast and less reliable
4. What keyword is used to skip the first row when loading a CSV file into a table? A. IGNORE FIRST ROW B. SKIP HEADER C. IGNORE 1 LINES D. SKIP 1 LINES
5. What statement should be used in MySQL to define an index? A. CREATE INDEX indexName ON tableName B. CREATE INDEX indexName ON tableName (column) C. CREATE INDEX ON tableName (column) D. CREATE INDEX ON tableName
6. What is the beginning of the statement used in MySQL to define a unique index? A. CREATE UNIQUE INDEX B. CREATE NOT NULL INDEX C. There is no direct statement for this; to indicate that the index is unique, the field must be defined as unique in the table definition D. None of the answers is correct
7. The use of indexes A. Can improve the performance of certain queries B. Always improves the performance of certain queries C. Rarely improves the performance of certain queries D. Influences database space usage more than performance
8. What statement should be used in MySQL to delete an index? A. DROP INDEX B. REMOVE INDEX C. DELETE INDEX D. DISABLE INDEX
9. Regarding databases, what does the XAMPP software include? A. A MySQL DBMS and an IDE to manage it B. A MariaDB DBMS and an IDE to manage it C. Exclusively a MariaDB DBMS D. Exclusively a MySQL DBMS
10. How are the ports of the DBMS configured in the XAMPP environment? A. The ports are predefined by default and cannot be modified through XAMPP B. Through the config.inc.php file C. Through the my.ini file D. Through the config.inc.php and my.ini files

in the Appendix A section). Once the theoretical part of the course is taught, the students take the same test during a final review class to evaluate the knowledge they have acquired up to that point in time. The students then start the practical computer assignment in groups of three people. In the assignment, students face the following tasks using MySQL as a database management system: the configuration and creation of a database; the realization of an the Extraction-Transformation-Load process from several MS Excel and XML files; the optimization of queries using indexes; the creation of users and provision of permissions; and backup and recovery operations. The students perform the assignment in six 2-h hands-on computer classes, in which the teacher accompany them through the provision of step-by-step tutorials and the individualized resolution of doubts. To complete the assignment, the students submit a group report in which the procedures used to solve the above-mentioned tasks must be explained. Once the assignment is completed, the students again take the knowledge test for the last time, as well as an individual practical computer exam in a 90 min session. In that exam, the students should solve problems such as the ones they face during practical computer classes, such as loading data into MySQL from an XML file or optimizing queries using indexes. It is also mentionable that during the exam, the students’ screen is monitored using the Veyon software 4.9 to ensure that they are using the allowed version of ChatGPT (i.e., 3.5) and not other more advanced paid versions. Lastly, the students are invited to participate in a questionnaire about the usage, reliability, and utility of the resources employed. Before submitting the questionnaire, students give their informed consent to use the collected information for research purposes.

During the assignment, students are free to use any resource they consider to complete the work: notes and tutorials provided by the teacher, explanations of the teacher or colleagues, MySQL official documentation, traditional websites (e.g., Stack Overflow and Google) and GenAI systems (specifically, ChatGPT v.3.5). The students are encouraged to use all possible resources, and in the practical sessions, the teacher exemplifies the utilization of these resources through step-by-step tutorials in which they solve some common problems using these resources. During the practical exam, students are free to use any resource they considered, except for messaging applications that would allow them to communicate synchronously with other people or non-free IAGen tools. Allowed resources include the assignment report, tutorials provided by the teacher, MySQL official documentation, traditional websites, and ChatGPT v.3.5.

2.4. Methods and Instruments

An essential data set collected in the case study is the students’ grades obtained in the individual exam, which is scored from 0 to 10. Moreover, additional data come from the results obtained in the tests performed during the learning experience: one test before starting the theoretical part of the course (test 1), another when the theoretical part of the course is finished and the practical part is about to start (test 2), and a last one when the practical part is finished (test 3). All tests contain the same 10 multiple-choice questions (see Table A1 in the Appendix A section), which are designed and validated by the course teacher. These tests do not count for the final grade, thus relaxing possible students under stress response and, consequently, avoiding undesired behavior. The test solution is not revealed until the end of the experience.

In addition, another important data set comprises the results of a questionnaire administered to collect students’ opinions about the resources used during the exam realization. The questionnaire comprises two sections. The first section includes questions about age, gender, classroom attendance, and prior experience with ChatGPT. The second section includes statements on the usage, reliability, and utility of the resources available to the students during the exam, as well as a question about the purpose for which they used ChatGPT (if used). Most of the questions, except for the ones about age and gender and the one on ChatGPT uses (in which the students selected options from a closed list), are rated on a Likert scale from one (nothing) to five (a lot). In the statements about the reliability and utility of the resources, students could also select the option ‘don’t know/no answer’. Questionnaire items are presented together with the results. The validity of the questionnaire content is checked through a review by experts in educational innovation, whereas its internal consistency type reliability is evaluated by means of Cronbach’s alpha, in which a value of 0.771 is obtained, indicating that the reliability is acceptable.

2.5. Data Analysis

The data collected are publicly available at the following link: https://doi.org/10.21950/XMROTF, accessed on 28 November 2024. The results of the student’s grades and the questionnaire are analyzed using two descriptive statistics: the mean (M) and the standard deviation (SD). In addition, inferential results are computed. To do so, the Kolmogorov–Smirnov normality test is conducted to check the normality of the data, which results in not normally distributed. Therefore, non-parametric statistical methods are used.

First, the Wilcoxon test [38] is used to make pairwise comparisons of the theoretical tests performed throughout the course. That is, to compare the results of test 1 with test 2, test 2 with test 3, and test 1 with test 3. The statistic used to measure the effect size of the differences analyzed is the correlation coefficient (r) [39], whose thresholds are as follows:

$$\{r\in \mathrm{I}\mathrm{R}|-1<r<1\}\left\{\begin{array}{cc}\left|r\right|<0.1\hfill & \mathrm{negligible}\mathrm{effect}\hfill \\ 0.1\le \left|r\right|<0.3\hfill & \mathrm{small}\mathrm{effect}\hfill \\ 0.3\le \left|r\right|<0.5\hfill & \mathrm{medium}\mathrm{effect}\hfill \\ 0.5\le \left|r\right|\hfill & \mathrm{large}\mathrm{effect}\hfill \end{array}\right.$$

(1)

Second, the Kruskal–Wallis test [40] is used to compare and find possible statistically significant differences when organizing students according to their grades. In this case, the Eta Squared coefficient (${\eta }^2$) is employed to study the effect size of these differences [39]. The thresholds of this coefficient are as follows:

$$\{{\eta }^2\in \mathrm{I}\mathrm{R}|0<{\eta }^2<1\}\left\{\begin{array}{cc}{\eta }^2<0.01\hfill & \mathrm{negligible}\mathrm{effect}\hfill \\ 0.01\le {\eta }^2<0.06\hfill & \mathrm{small}\mathrm{effect}\hfill \\ 0.06\le {\eta }^2<0.14\hfill & \mathrm{medium}\mathrm{effect}\hfill \\ 0.14\le {\eta }^2\hfill & \mathrm{large}\mathrm{effect}\hfill \end{array}\right.$$

(2)

Lastly, the Spearman correlation test [41] is used to correlate the student’s grades with the student’s responses to the questionnaire items. Regarding the Spearman correlation coefficient Rho ($\rho$), a positive value means a positive correlation, and a negative value means a negative correlation. The thresholds for this coefficient are as follows:

$$\{\rho \in \mathrm{I}\mathrm{R}|-1<\rho <1\}\left\{\begin{array}{cc}\left|\rho \right|<0.1\hfill & \mathrm{no}\mathrm{correlation}\hfill \\ 0.1\le \left|\rho \right|<0.3\hfill & \mathrm{low}\mathrm{correlation}\hfill \\ 0.3\le \left|\rho \right|<0.5\hfill & \mathrm{medium}\mathrm{correlation}\hfill \\ 0.5\le \left|\rho \right|\hfill & \mathrm{high}\mathrm{correlation}\hfill \end{array}\right.$$

(3)

3. Results

3.1. Student’s Exam Grades

The mean grade is 4.85 with an SD of 3.22. Of the 40 students, 18 failed the exam (grade less than 5), 7 obtained a pass grade (grade between 5 and 7), and 15 obtained an outstanding grade (grade greater than 7).

3.2. Student’s Test Results

Throughout the experience, the students complete a test several times. As indicated above, one test is performed before starting the theoretical part of the course (test 1), another when the theoretical part of the course is finished and the practical part is about to start (test 2), and the last one when the practical part is finished (test 3).

Table 1. Student’s test results.

	Test 1	Test 2	Test 3
M (SD)	0.60 (0.71)	4.73 (3.01)	8.43 (2.04)

depicts the results.

Pairwise differences are compared using the Wilcoxon test to determine whether the differences are statistically significant, and the correlation coefficient r is calculated to observe the effect size of the differences.

Table 2. Comparison of student’s test results.

Comparison	Significance (p-Value)	Effect Size (r)
Test 1 vs. Test 2	<0.001	0.68
Test 2 vs. Test 3	<0.001	0.58
Test 1 vs. Test 3	<0.001	0.93

shows these inferential results.

3.3. Usage and Perceived Reliability and Utility of Resources

The results obtained from the questionnaire show the usage, reliability, and utility of some learning resources available to perform the individual exam. The specific questions are as follows: ‘To what extent did you use the following resources?’ (usage), ’How reliable did you find the employed resources?’ (reliability), and ‘How useful did yo find the following resources?’ (utility). As mentioned above, the evaluated resources are as follows: assignment report, assignment materials, MySQL official documentation, traditional internet resources, and ChatGPT.

Table 3. Usage, reliability, and utility of resources during the exam.

	Item	Usage M (SD)	Reliability M (SD)	Utility M (SD)
1	Assignment report	4.45 (0.88)	4.63 (0.59)	4.50 (0.93)
2	Assignment materials (i.e., teacher tutorials)	2.98 (1.37)	4.16 (1.02)	3.36 (1.22)
3	MySQL official documentation	2.20 (1.26)	4.00 (1.20)	3.04 (1.34)
4	Traditional internet resources (Google, Stack Overflow, etc.)	2.43 (1.36)	3.36 (1.19)	3.00 (1.19)
5	ChatGPTv3.5	3.13 (1.40)	3.39 (1.12)	3.55 (1.35)

shows the results.

Moreover, the questionnaire also includes a specific question about the purposes for which the students used ChatGPT during the exam. Figure 1 depicts these results, ordered by the number of occurrences. Note that 8 of 40 students (20%) did not use ChatGPT.

3.4. Student’s Test Results Grouped by the Student’s Exam Grades

The students can be grouped according to the grade obtained in the practical exam, which is scaled in three categories: fail (grade < 5), pass (5 ≤ grade < 7), and outstanding (grade ≥ 7). This leads to three groups: Group 1 (grade = fail, N = 18), Group 2 (grade = pass, N = 7), and Group 3 (grade = outstanding, N = 15).

Table 4. Student’s test results (grouped by grades).

Test	Group 1 M (SD)	Group 2 M (SD)	Group 3 M (SD)	p-Value (${\mathit{\eta }}^2$)
Test 1	0.61 (0.70)	0.57 (0.79)	0.60 (0.74)	0.979 (0.00)
Test 2 *	3.05 (2.26)	5.85 (3.98)	6.20 (2.40)	0.008 (0.259)
Test 3	7.83 (2.48)	8.71 (1.60)	9.00 (1.46)	0.38 (0.073)

shows the test results analyzed by groups. In this case, the statistical significance of the differences is analyzed using the Kruskal–Wallis test, and the effect size is analyzed by using the eta squared coefficient (${\eta }^2$). Statistically significant differences appear in the following table with a ‘*’ symbol.

3.5. Classroom Attendance, Prior Experience with ChatGPT and Prompting Skills

The questionnaire also includes three questions to ask the students about their classroom attendance, prior use of ChatGPT, and their prompting skills.

Table 5. Classroom attendance, prior experience with ChatGPT, and prompting skills.

Item	Total M (SD)	Group 1 M (SD)	Group 2 M (SD)	Group 3 M (SD)
Classroom attendance	4.30 (0.94)	4.39 (0.98)	4.14 (0.69)	4.27 (1.03)
ChatGPT prior experience	3.70 (0.82)	3.72 (0.83)	3.57 (0.98)	3.73 (0.80)
Prompting skills	3.55 (1.01)	3.56 (1.04)	3.71 (0.95)	3.47 (1.06)

depicts the overall results in this regard (see column total), as well as the results analyzed by the aforementioned groups (see the remaining columns). The differences among the groups are analyzed by using the Kruskal–Wallis test, but no significant differences are found.

3.6. Usage and Perceived Reliability and Utility of Resources Grouped by Student’s Grades

As mentioned before, the students can be grouped by their grades according to the scale previously mentioned (fail, pass and outstanding), leading to three groups. Figure 2 shows the results regarding the item ‘I did not use ChatGPT during the exam’ analyzed by groups.

Table 6. Usage of resources during the exam (grouped by grades).

	Item	Group 1 M (SD)	Group 2 M (SD)	Group 3 M (SD)	p-Value (${\mathit{\eta }}^2$)
1	Assignment report	4.22 (1.11)	4.29 (0.76)	4.80 (0.41)	0.14 (0.099)
2	Assignment materials (i.e., teacher tutorials) *	3.67 (1.14)	3.67 (1.14)	2.20 (1.21)	0.008 (0.243)
3	MySQL official documentation *	2.61 (1.33)	2.61 (1.33)	1.53 (0.74)	0.049 (0.171)
4	Traditional internet resources (Google, Stack Overflow, etc.) *	2.83 (1.29)	2.71 (1.11)	1.80 (1.37)	0.045 (0.132)
5	ChatGPT v3.5 *	2.44 (1.25)	3.00 (1.29)	4.00 (1.20)	0.007 (0.261)

,

Table 7. Perceived reliability of resources during the exam (grouped by grades).

	Item	Group 1 M (SD)	Group 2 M (SD)	Group 3 M (SD)	p-Value (${\mathit{\eta }}^2$)
1	Assignment report	4.56 (0.70)	4.57 (0.53)	4.73 (0.46)	0.74 (0.021)
2	Assignment materials (i.e., teacher tutorials)	4.35 (0.79)	4.40 (0.89)	3.70 (1.34)	0.37 (0.094)
3	MySQL official documentation	4.14 (1.10)	3.50 (1.52)	4.17 (1.17)	0.55 (0.054)
4	Traditional internet resources (Google, Stack Overflow, etc.)	3.50 (0.76)	3.00 (1.41)	3.43 (1.72)	0.70 (0.032)
5	ChatGPT v3.5 *	2.75 (0.75)	3.67 (1.21)	3.80 (1.15)	0.02 (0.198)

and

Table 8. Perceived utility of resources during the exam (grouped by grades).

	Item	Group 1 M (SD)	Group 2 M (SD)	Group 3 M (SD)	p-Value (${\mathit{\eta }}^2$)
1	Assignment report	4.28 (1.23)	4.71 (0.29)	4.67 (0.62)	0.76 (0.048)
2	Assignment materials (i.e., teacher tutorials) *	3.76 (0.90)	4.20 (0.84)	2.36 (1.21)	0.006 (0.362)
3	MySQL official documentation	3.29 (1.14)	3.17 (1.33)	2.33 (1.75)	0.46 (0.088)
4	Traditional internet resources (Google, Stack Overflow, etc.)	3.29 (0.99)	2.71 (1.11)	2.71 (1.60)	0.36 (0.060)
5	ChatGPT v3.5 *	2.92 (1.31)	3.17 (1.17)	4.20 (1.21)	0.02 (0.207)

depict, respectively, the usage, reliability, and utility of the available resources during the exam according to the defined groups. For each group, the mean and standard deviation (in parentheses) are displayed. Moreover, the statistical significance of the differences is analyzed using the Kruskal–Wallis test, and the effect size is analyzed using the eta squared coefficient (${\eta }^2$). Statistically significant differences appear in the following tables with a ‘*’ symbol.

3.7. Relation Between the Student’s Grades and the Usage, Reliability, and Utility of Resources

The correlations using the Spearman technique are calculated in order to know how the student’s grades are related to the usage and perceived reliability and utility of the available resources during the completion of the exam.

Table 9. Correlation between the student’s grade and the usage, reliability and utility of resources during the exam realization.

	Item	Usage $\mathit{\rho }$ (p-Value)	Reliability $\mathit{\rho }$ (p-Value)	Utility $\mathit{\rho }$ (p-Value)
1	Assignment report	0.22 (0.16)	0.07 (0.66)	0.07 (0.66)
2	Assignment materials (i.e., teacher tutorials)	−0.51 (<0.001) *	−0.28 (0.11)	−0.39 (0.02)
3	MySQL official documentation	−0.29 (0.06)	−0.08 (0.65)	−0.24 (0.20)
4	Traditional internet resources (Google, Stack Overflow, etc.)	−0.20 (0.19)	−0.10 (0.61)	−0.09 (0.64)
5	ChatGPT v3.5	0.45 (0.003) *	0.42 (0.01) *	0.48 (0.005) *

shows the results of these correlations, which are represented using the Spearman value (Rho, $\rho$) and the p-value. Statistically significant correlations appear in the following table with a ‘*’ symbol. Please note that if a student does not use a learning resource, the rating of its use would be the lowest (1 on the employed Likert scale), while the usefulness or reliability would be rated NS/NC. In the calculation of the correlations, these data with NS/NC ratings are not considered.

4. Discussion

4.1. RQ1. How Widespread Is the Use of ChatGPT Among Students to Learn Database Administration, and What Is the Perceived Reliability and Usefulness of This Educational Resource Compared to Other Traditional Resources?

The results reported for the presented case study (see Table 3) indicate that ChatGPT usage is moderate (3.13 out of 5), but it narrowly outperforms other traditional resources such as MySQL official documentation (2.20), and Stack Overflow posts or googling (2.43) in solving practical database management problems. This result is striking considering that in a similar case study conducted one year ago [42], ChatGPT was less used than the aforementioned resources. This indicates, in line with other previous studies [1,2], that GenAI tools are becoming increasingly widespread in education and the field of computer science education is no exception.

However, the use of ChatGPT is not too extensive compared to the resource most used by the students: the report that they themselves had prepared during the practical sessions (4.45 out of 5). This report is the result of their work in the practical classes for weeks during which, in addition to the aforementioned resources, they also had the support of the teacher (providing step-by-step tutorials and resolving individual inquiries) and their colleagues in the practice group to solve practical problems as they arose.

Regarding the reliability of the digital resources used, it should be noted that the students perceived ChatGPT (3.39 out of 5) as less reliable than the tutorials provided by the teacher (4.16), MySQL official documentation (4.00), or their own report (4.63). This is to be expected, as ChatGPT itself warns you that its answers may not be true, and it is common to make ChatGPT (especially the free version 3.5, the one that the students used) fall into hallucinations leading to wrong answers [43]. Despite this, the students considered ChatGPT to be more useful (3.55 out of 5) than teacher tutorials (3.36), MySQL official documentation (3.04), and Stack Overflow or googling (3.00). This is understandable since the use of the other tools requires digging into the information and looking for an answer that ChatGPT can give you directly if you ask it properly. Nevertheless, the most useful resource for the students is the report they prepared during the classes before the exam (4.50).

Furthermore, it is interesting to explore the specific usage of ChatGPT to solve database administration problems (see Figure 1): 80% of the students declared that they used this tool and used it mainly to look for alternatives or improvements to certain SQL statements, to solve problems with a SQL statement, and to know about the syntax or examples of certain SQL statements. For example, ChatGPT proved to be quite useful for students who wanted to learn details about SQL statements for creating users or granting permissions but it was not very useful to identify the SQL statements necessary to load information in a MySQL database from an MS Excel or an XML file or to solve problems during the loading process, such as incompatibility between the character set of the data. In these cases, the solution usually came from a Socratic question of the teacher, a targeted instruction from a classmate, or other traditional sources.

Lastly, note that the use of ChatGPT among students is remarkably uneven. It is striking to observe the distribution of ChatGPT usage among students if they are categorized into the aforementioned groups (see Figure 2). While 100% of the students who obtained an outstanding score (grade higher than 7) in the practical exam used ChatGPT during the exam, only 61% of the students who failed used it. However, we must analyze more thoroughly the degree of use of ChatGPT during the exam (see Table 6), as well as the reliability (see Table 7) and the utility (see Table 8) perceived by the students. It can be seen that the students who obtained the best marks used ChatGPT intensively (4 out of 5), significantly more than the other students. In the same vein, these outstanding students also perceived ChatGPT to be much more reliable (3.80) than students who failed the exam (2.75), finding here a statistically significant difference (p < 0.05) with a large effect size (${\eta }^2$ > 0.14). Another statistically significant difference (p < 0.05) with large effect size (${\eta }^2$ > 0.14) was found by comparing the usefulness of ChatGPT perceived by outstanding students (4.20) with those who failed the exam (2.92). Therefore, the group of students with the best scores in the exam was the one who used ChatGPT the most and had the best perception of its reliability and usefulness. This may be due to multiple reasons that are explored in the following research question.

4.2. RQ2. How Does Prior Knowledge About Database Administration Affect the Use of ChatGPT to Solve Database Administration Problems?

First, it is relevant to observe the progression of knowledge acquired by the students throughout the course (see Table 1). It can be seen that at the beginning of the course, the students knew practically nothing about database administration (Test 1, 0.60 out of 10), that once the theoretical part of the course was finished, they had acquired several theoretical concepts (Test 2, 4.73), and that at the end of the practical part of the course, they had acquired almost all of them (Test 3, 8.43). The inferential comparative results calculated in this regard (see Table 2) indicate that the differences in the results between the tests performed at the different times of the course are statistically significant (p < 0.001) and have a large effect size (r > 0.50).

However, the student performance is very uneven. In fact, as indicated above, of the 40 students involved in this case study, 18 finally obtained a grade lower than 5 in the practical exam (Group 1), 7 students obtained a grade between 5 and 7 (Group 2), and 15 students obtained a grade higher than 7 (Group 3). We must analyze the progress in the acquisition of knowledge of these three groups of students. The descriptive and inferential results reported (see Table 4) indicate that at the beginning of the course, the students of the three groups started from a very similar base (0.61 vs. 0.57 vs. 0.61) but the students’ performance in the theoretical part of the course (test 2) is very unequal since it can be observed that the group of students who ended up failing the practical exam obtained a significantly worse grade (3.05) than the groups of students who finally approved the practical exam with a grade of pass (5.85) or outstanding (6.20). This difference in knowledge allowed the students to approach the practical work very differently because the students who already had a stronger theoretical basis could face the practical problems posed by the practical assignment with much more criteria and settled fundamentals than the others. That may have favored the use of tools such as ChatGPT. At the end of the course, with the practical work completed, it seems (Test 3) that the relative difference in terms of acquired theoretical knowledge was reduced between the three groups but there was still a notable difference with a medium effect size (7.83 vs. 8.71 vs. 9.00).

This difference in theoretical knowledge before facing the practical work may explain the more intensive use of ChatGPT and the better perceptions about ChatGPT by the students who ended with an outstanding grade discussed in the previous RQ. This is somewhat expected since, as indicated by previous studies [36], a person with more knowledge about a topic is likely to be better able to use GenAI tools since he/she knows exactly what to ask about [44], knows how to continue a conversation with an IAGen and inquire more about a topic, knows how to identify hallucinations of the IAGen, and knows how to redirect the conversation [45]. So, we deem that the difference in prior knowledge is a factor that led these students to use ChatGPT to a greater extent and with greater success during the exam. But could there be other factors that explain this? Two elements that could do so are explored below: classroom attendance and prior experience with ChatGPT and prompting skills.

First, classroom attendance could be a key element in successfully performing the practical work and also in properly using tools such as ChatGPT since during the practical sessions, the teacher provided examples of how to use ChatGPT to solve database administration problems such as those faced by the students. Therefore, it could be inferred that the group of students who ended up failing the exam were the ones who missed more classes, but that was not the case. In fact, as shown in Table 5, this group of students attended classes (4.39 out of 5) slightly more than the group of students who obtained a pass grade (4.14) or outstanding (4.27). However, these differences are minimal and are not statistically significant.

Second, prior experience with ChatGPT and prompting skills seems to be another key element that can determine the success of using this tool. It can be observed (see Table 5) that the students reported having a medium–high degree of prior experience with ChatGPT (3.70 out of 5) and rated their prompting skills as reasonably good (3.55). Nevertheless, it is astounding that the different groups declared similar levels of prior experience with ChatGPT and self-perceived that they had similar prompting abilities, and no significant differences were found in this respect.

Therefore, it seems that in the case study under analysis, neither classroom attendance nor prior use of ChatGPT and self-perceived prompting skills caused certain students to use ChatGPT more and to have a better perception of it. Therefore, we deem that it is reasonable to think that prior knowledge of the subject before the practical work is what led to a more intensive and adequate use of ChatGPT by students.

4.3. RQ3. Is There a Relationship Between the Usage of ChatGPT and the Success in Solving Practical Database Administration Problems?

The results discussed previously suggest that there is some relationship between the use of ChatGPT and the final grade of the students on the practical exam, in which they had to solve computer database administration problems. To explore this further, it is useful to analyze the results presented in Table 9, which shows the correlations between the student’s grade and the use and perception of the different resources available to him or her to take the exam. It can be observed that there are statistically significant medium-to-high positive correlations between the students’ grade and degree of use ($\rho$ = 0.45, p-value = 0.003), and perceived reliability ($\rho$ = 0.42, p-value = 0.01) and utility ($\rho$ = 0.48, p-value = 0.005) of ChatGPT. A positive correlation of student grade with assignment report use, reliability and utility is also found, but it is not as high or statistically significant. In addition, a significant, negative and high correlation of grade is found with the use of teacher tutorials during the exam ($\rho$ = −0.51, p-value < 0.001), which is logical since these tutorials were introductory and served to lay certain foundations, but not to solve advanced problems posed by the practical exam. In the case of the rest of the learning resources, we find negative correlations with the final grade, but they are low correlations (Rho < 0.30) without statistical significance.

Therefore, the posed RQ can be answered positively, and it can be affirmed that there is a positive relationship between the use of ChatGPT and the success in solving practical database administration problems. We consider GenAI tools such as ChatGPT to be helpful resources to assist in the resolution of database administration problems, and to have a positive impact on student performance. This is partially consistent with the conclusions of [46], where the usage of ChatGPT significantly improved the quality and originality of the student task resolution. Our conclusions are also partially consistent with those reported in the aforementioned work of [44], which demonstrated that professional software engineers improve their skills by prompting them not to write code but to solve higher-order knowledge tasks. Indeed, as in the line of our work, Ref. [47] showed that the knowledge base of the subject significantly affects the effectiveness of learners in applying ChatGPT to solve complex problems.

Lastly, it should be mentioned that the student’s final grade may have relations with other factors such as classroom attendance or prior ChatGPT use and prompting skills, but as discussed in the previous RQ, no differences in such factors are identified among the different groups of students. However, teacher guidance during practical sessions was necessary to show students pieces of prompt engineering with the intention of solving problems step by step [43] since it contributes to improving student skills both with the use of ChatGPT and with the subject [48]. There may also be other unexplored factors related to student performance on the exam, such as time and stress management skills, that are very important when taking an exam. In any case, what has been found in this case study is a clear relationship between the use of ChatGPT and student performance.

5. Conclusions

This article has presented an empirical study to explore the usage of ChatGPT to learn about database administration in computer science education. We found that ChatGPT was moderately used, but it was more used than traditional internet learning resources, and a trend of increasing use could be observed compared to the previous year. We also found uneven usage by students and that the students who end up earning better grades were the ones who used it more and had better perceptions about ChatGPT. In fact, positive correlations were found between the use of this tool and the success of solving database administration problems during a practical exam. Lastly, we found that one of the elements that could encourage students to use this tool intensively and productively was the theoretical background they had about database administration.

Therefore, it can be concluded that ChatGPT can be an excellent educational instrument in the context of database administration and that to use it properly, it is necessary for students to have a solid theoretical basis. However, as mentioned in the Introduction, it should be noted that GenAI tools such as ChatGPT have a number of limitations to consider: the possibility of hallucinating, the lack of critical thinking that could be generated in the GenAI users, and even prohibitions on the use of these tools in the process of software development by contractors.

In addition, during the execution of this case study, we have found that prompting skills are also important, and teachers can use a guided practice strategy, where prompting and guided step-by-step practice are employed to train these skills as well as to increase the student’s theoretical background. Training students in the use of GenAI tools like ChatGPT is key to prevent the appearance of new digital trenches between students with solid knowledge and prompting skills that can further increase their performance through the proper use of GenAI tools and students with a lower degree of the fundamentals and worse prompting skills, who lag behind and will obtain even worse results.

The presented conclusions are strongly connected to computer science education since they are based on a case study in that context, but we deem that they are fully transferable to any engineering or science discipline in which the practical mastery of technologies and problem solving play a key role.

Lastly, some limitations and future work are presented. First, the sample size is only 40 students, so the conclusions reported here should be taken with caution. Furthermore, there are many difficulties associated with the generalization of single case studies like the presented [49], so more studies like this should be conducted to consolidate our findings. Once this occurs and the presented framework has been applied several times, we could provide a more reliable and generalizable mathematical model. Second, the gender distribution is unbalanced, which is common in computer science studies, and unfortunately we do not have a way to solve this limitation in the study. Third, the study design precludes the assertion that ChatGPT caused higher grades. The conduction of a randomized controlled trial in which there was an experimental group with access to ChatGPT and a control group without access to ChatGPT would allow one to isolate the ChatGPT variable and assert cause-and-effect relationships. However, although this is very desirable from a research point of view, from a teaching perspective it is complicated, as it violates the principle of student equity. Fourth, the conclusions reported are based on various data, including those obtained through the questionnaire. This instrument provides self-reported data that can influence responses due to social desirability bias and inaccurate self-reporting. In detail, the prompting skills of the students are measured using this self-perception questionnaire, rather than more objective tests. In the future, the prompting skills of the students could be measured by analyzing their prompts. Lastly, conducting studies in the future using qualitative data or assessing long-term knowledge retention would provide a more complete picture of ChatGPT’s impact on learning.