Towards AI Dashboards in Financial Services: Design and Implementation of an AI Development Dashboard for Credit Assessment

Abstract

Financial institutions are increasingly turning to artificial intelligence (AI) to improve their decision-making processes and gain a competitive edge. Due to the iterative process of AI development, it is mandatory to have a structured process in place, from the design to the deployment of AI-based services in the finance industry. This process must include the required validation and coordination with regulatory authorities. An appropriate dashboard can help to shape and structure the process of model development, e.g., for credit assessment in the finance industry. In addition, the analysis of datasets must be included as an important part of the dashboard to understand the reasons for changes in model performance. Furthermore, a dashboard can undertake documentation tasks to make the process of model development traceable, explainable, and transparent, as required by regulatory authorities in the finance industry. This can offer a comprehensive solution for financial companies to optimize their models, improve regulatory compliance, and ultimately foster sustainable growth in an increasingly competitive market. In this study, we investigate the requirements and provide a prototypical dashboard to create, manage, compare, and validate AI models to be used in the credit assessment of private customers.

1. Introduction

For the assessment of financial credit risk, the use of AI to accelerate, automate, and improve the credit assessment process is becoming increasingly attractive [1]. Moreover, increased computational power has made it possible to analyze vast amounts of data and to recognize behavioral patterns relating to loans. The example of credit risk assessment illustrates how AI must be utilized in the financial services industry, demonstrating the necessity for precision, efficiency, and the ability to handle large datasets to identify and evaluate potential risks accurately. Therefore, the integration of AI in financial services has become a transformative force in research and practice [2]. However, previous research has primarily focused on testing AI techniques to increase the predictive accuracy of credit assessment [3,4] (Appendix B). However, the integration of AI is leading to a paradigm shift in the way that financial institutions should develop, maintain, and operate AI and concurrently cooperate with regulators [5,6]. Therefore, in the long term, it is important to derive concepts or dashboards for model development to operate, monitor, and maintain AI models with suitable and compliant processes [7].

These challenges to the use of AI in financial services are increasingly being addressed by politicians, as they attempt to define regulations for the use of AI to promote trust, manage risks appropriately, and encourage AI innovations [8,9]. According to the European Commission’s “AI Act” [10], the entire process of model creation, including the data used and all measures taken, must comply with various quality guidelines and must be documented in detail. For example, the processes of data cleaning and preprocessing must be fully documented to prove that the underlying dataset has been prepared appropriately and compliantly with the AI Act, e.g., taking account of biases in the dataset [10]. Moreover, the problem of masking the internal decision-making process is addressed by regulatory requirements so that the explainability, interpretability, traceability, transparency, and security of AI-based services is increased [9,11,12]. To address the challenges identified in the integration of AI for credit risk assessment, various commercial tools have been developed to facilitate the creation and management of AI-driven credit risk dashboards. Leading examples, such as FICO, Zest AI, and CreditVidya, provide comprehensive analytics and visualization capabilities. However, financial companies must define iterative and compliant processes in model development for regulatory adherence in financial services, ensuring accuracy and transparency. Furthermore, all these aspects must be comprehensively considered during the design of AI-based services and associated processes to fulfill regulatory requirements.

Furthermore, previous research on AI in credit assessment indicates that AI-based models can deliver more accurate and performant results than traditional statistical models [7,13,14]. However, AI models add complexity and require an iterative approach over the long term. This creates a conflict where higher performance often comes at the cost of lower explainability, and vice versa. The development of these models corresponds to a data mining process and is associated with many phases and steps, ranging from the data preprocessing to the implementation of the model. Such a process is highly time-consuming and can take between 9 and 18 months [15,16,17,18]. Moreover, these datasets face challenges such as an uneven distribution of “good” and “bad” loans, and a time lag between data collection and model application.

Problems with the datasets can lead to a decrease in performance and intensify over time, a phenomenon known as concept shift or model shift [19]. Serious incidents like the 2007 financial crisis and the COVID-19 pandemic significantly trigger such conceptual shifts, requiring banks to continually validate and adjust their models to maintain functionality and accuracy [20,21,22]. This involves regular and ad-hoc validations, potentially leading to recalibrations or new models [21]. Furthermore, standardized procedures and comprehensive documentation are essential for structured control by financial companies and regulators [21]. Therefore, AI dashboards can assist in these iterative analyses by systematically evaluating datasets, developing and monitoring models, and enabling swift actions to ensure system accuracy for customer safety.

Moreover, mandatory cooperation with supervisory authorities and the lack of structure in model development and validation need to be addressed, as scoring model evaluations are often inadequate. To effectively analyze and monitor datasets and AI models, it is essential to structure and foster iterative processes like data analysis and model evaluation and comparison. Identifying and addressing concept-shift issues requires analyzing the performance of existing or new AI models with incoming data. A dashboard supporting these iterative analyses and documentation can enhance the development, maintenance, and evaluation of AI models [13]. This study addresses the research gap in structuring and systematizing the development, training, testing, comparison, evaluation, and deployment of AI models for credit assessment. The following research questions (RQs) aim to bring more structure to the field through automated documentation and clearly defined procedures:

• RQ₁: What are the requirements for an AI dashboard to develop and maintain AI models for credit assessment?
• RQ₂: How should an AI dashboard for credit assessment be designed and the model development process structured?

In the following, we first introduce the related research into AI-based services. Subsequently, we present the underlying methodology of the conducted design science research framework [23] and the employed cross-industry standard process for data mining (CRISP-DM) [24]. We then derive the requirements before presenting the design and development process leading to the final dashboard design. After the demonstration of the artifact, we discuss our results, mention limitations, and state a conclusion with an outlook on future research.

2. Related Research for the Use of AI in the Finance Industry

The accurate assessment of creditworthiness, prediction of bankruptcy, and detection or prevention of fraud are crucial for informed financial decision making, influencing access to loans, insurance, and other products. Moreover, while many of these processes have traditionally relied on established statistical models, the rise of AI has introduced a new paradigm with significant implications. For example, rule-based traditional models for credit assessment such as FICO scores are based on factors that are easy to understand, such as income, debt, and payment history. Thanks to this transparency, borrowers can understand how their score is calculated and can take action to improve it [25]. Therefore, the rule-based conventional models adhere to established regulations and guidelines and ensure fairness and consistency in credit assessment, with the logic behind them being relatively simple and making it easier for human analysts to understand and explain decisions. However, the governance of data-driven AI models strongly differs from the conventional models and requires a customized governance process to focus more on AI-specific criteria and risks [26,27]. Understanding trained AI models can be significantly enhanced through the extraction of patterns from datasets. The European Commission’s Artificial Intelligence Act highlights the necessity of distinguishing between so-called Weak AI, which focuses on specific tasks, and Strong AI, which exhibits more generalized cognitive capabilities [28].

Ensuring the trustworthiness of AI requires that datasets are free from biases and protect individual privacy. Adhering to these principles not only aids in the development of robust and versatile AI systems but also ensures they are ethical and secure, fostering greater public confidence and societal advantages. Furthermore, the problem of customized governance has also been considered by regulatory authorities in many European countries [20,21,29,30,31,32,33]. For example, the French Prudential Supervision and Resolution Authority (ACPR) published the first discussion paper in 2020, considering the issues with explainability and governance of AI [29]. ACPR defined four evaluation principles to design and develop AI algorithms and models: appropriate data management, performance, stability, and explainability. These four principles include the required structure and characteristics of AI-based services in financial services that must be defined, developed, and deployed in a compliant structure to ensure customers’ safety. However, due to the potential to increase automation in financial services with AI, all underlying and related processes and systems to develop and maintain these AI models must also be accelerated [34]. Therefore, the complexity of implementation and quality assurance issues become major obstacles preventing financial institutions from adopting AI-based services [34].

Thus, a balance must be struck between innovation and risk when using AI in financial services. Accordingly, it is necessary to develop not only accurate AI models but also systems or dashboards to maintain, develop, and monitor these models. Nevertheless, the existing literature aims to develop suitable and highly accurate models for datasets [35]. These approaches cannot guide the financial industry forward if there is no combination of well-defined systems and processes to maintain the models to ensure continued system security, as an AI model cannot operate properly forever [36]. In addition, the operational quality and robustness of the models must be monitored and validated regularly [10]. In case of adjustments required due to model or data drift, e.g., caused by changing economic conditions, financial companies or providers of these AI-based services must be equipped with potential solutions through increased automation, integration, and mitigation measures [26]. However, all the specific governance needs vary depending on the use case [37].

Furthermore, existing regulatory bases and recommendations from the OECD [38] focus on responsibility for trustworthy AI and require proactivity and sustainable development. Therefore, it is necessary to foster a regulatory environment that supports a flexible transition from research to the development, deployment, and operation of AI-based services [9]. Consideration of a controlled environment, e.g., an AI dashboard, is necessary to be able to test and scale up the proposed models [37,38]. Moreover, this interactivity between financial firms and regulators needs to be improved to promote entrepreneurship and productivity and ensure the benefits of AI [29,38,39].

3. Research Design

To target the research gap outlined in the first section, we employed a problem-oriented design science research approach (DSR) according to Peffers et al. [23] in combination with CRISP-DM [24] to structure and support the process of model development for credit assessment with an AI dashboard. DSR helps us to structure the research process while identifying the problem, analyzing requirements, and developing artifacts (Figure 1). Moreover, due to the structure of CRISP-DM across industries for data mining processes, it provides a fundamental and appropriate basis on which to structure the process of model development and model evaluation for credit assessment with an associated understanding of business and data.

We followed the CRISP-DM approach to implement a standardized, non-iterative process that significantly reduces lengthy iterations. CRISP-DM’s established structure allows for extensions and simplifies the model development process for an AI dashboard for financial companies, developers, and regulators. The first step, business understanding, determined the requirements for an AI dashboard in credit assessment. As already outlined, with CRISP-DM, it is necessary to investigate the underlying dataset for a practical and useful orientation to propose an appropriate list of requirements.

The process of requirements analysis, which is based on four steps of the CRSIP-DM, is described in Section 4.1 (Figure 2). First, we investigate the requirements for business and data understanding based on the credit assessment dataset. Second, we identify the requirements for data preprocessing and preparation including data analysis. Third, we gather the requirements for modeling to support the process of model development on the AI dashboard. Lastly, we identify the requirements for the evaluation and comparison of models.

In Section 4.2, we derive the functionalities to fulfill the identified requirements based on the use-case analysis. Moreover, due to the regulations in the finance industry, it is necessary to analyze and consider regulatory requirements to select appropriate modeling techniques for an AI dashboard (Section 4.3). Based on the analysis of the requirements, use cases, and regulatory basis, we designed and developed the proposed artifact for the development and maintenance of AI models, as described in Section 4.4. As the last step of problem-centered DSR, the demonstration of the artifact “AIDash” is described in Section 4.5, using an exemplary scenario in which a new model for credit assessment is developed and evaluated.

4. AI Dashboard for Model Development in Credit Assessment

We start with analysis of the problem and requirements, to structure the development process of the meta-artifact to support the iterative processes of data understanding, data preprocessing, model development, and model evaluation based on CRISP-DM.

4.1. Problem Identification and Requirement Analysis

For the development of an artifact for credit assessment, it is necessary to consider the existing approaches. Therefore, we conducted a structured literature review based on the methodological guidelines of Cooper [40] and vom Brocke et al. [41]. The search was conducted in established scientific databases JSTOR, AIS Electronic Library, Emerald Insight, and IEEE Xplore with the following search terms: “Credit Scoring”, “Credit Assessment”, “Artificial Intelligence”, and “Machine Learning”, from 2014 onwards. We initially found 1301 publications, of which 61 appeared to be relevant for AI-based credit assessment after analyzing the titles and abstracts of the accessible publications (Appendix B). Relevant articles were those that dealt with AI-based credit assessment, dashboards for model development, provided literature reviews or interview studies, or considered respective regulations.

Based on the conducted literature review, we first examined the existing studies dealing with AI-based credit assessment. As identified from the literature review, CRISP-DM can help to structure the process of model development for credit assessment. Here, however, it is not possible to cover all the phases of CRISP-DM. In this study, the phases of business understanding, and deployment were not in our focus, as they set the prerequisites for the planned dashboard and require further strategic decisions depending on the companies’ service infrastructure (

Table 1. CRISP-DM process model.

Business Understanding	Data Understanding	Data Preparation	Modeling	Evaluation	Deployment
Determine business objectives	Collect initial data	Select data	Select modeling techniques	Evaluate results	Plan deployment
Assess situation	Describe data	Clean data	Generate test design	Review process	Plan monitoring and maintenance
Determine data mining goals	Explore data	Construct data	Build model	Determine Next Steps	Produce final report
Produce project plan	Verify data quality	Integrate data	Assess model		Review project
		Format data

). In the following, we go through the four phases of CRISP-DM, conduct a requirement analysis for credit assessment, and summarize the gathered requirements and derived functionalities in a table.

4.1.1. Data Understanding of Credit Assessment

Datasets to train and build AI models consist of credit applications with positive and negative decisions based on several features such as previous payment behavior, deposits in bank accounts, bonds, and duration of employment. For this study, we used an exemplary dataset from Fahrmeir and Hamerle [42] including 1095 credit applications from private customers with 21 features, 300 rejections, and 795 approvals (Appendix A). The CRISP-DM approach consists of multiple steps distributed over phases. The first step of data understanding begins with the initial collection of data and includes functions to gain first insights about the dataset [24]. This involves determining the quality of the data, gaining initial impressions, and/or recognizing groups in the dataset from which hypotheses can later be formed. We first started gathering requirements from the second step of CRISP-DM, i.e., data understanding, for the analysis and management of datasets for credit assessment. The ability to manage datasets is crucial, as all subsequent steps are based on this data (R_DU 1). This step also includes loading (or uploading) the dataset into the dashboard, which enables the available dataset for analysis (F_1.1). Moreover, to work and integrate the datasets in the next steps, the prototype should provide an overview of the existing datasets [24]. As the datasets are possibly available in different formats, it is necessary to have the option of changing the parameters for import so that datasets can be integrated homogeneously and thus processed consistently (

Table 2. List of requirements and derived functionalities from CRISP-DM for the artifact.

Block	Requirements	Function Identifier	Derived Functions for Credit Assessment
Data Understanding (RDU)	RDU 1 Data collection	F1	Managing datasets
		F1.1	Uploading datasets
		F1.2	Changing parameters for import
	RDU 2 Data description	F1.3	Describing datasets
	RDU 3 Explore data	F2	Analyzing datasets
		F2.1	Plotting datasets
		F2.2	Comparing datasets
	RDU 4 Data assessment	-	-
Data Preprocessing (RDP)	RDP 1 Data selection	F1.4	Managing features
	RDP 2 Data cleaning	F1.2	Changing parameters for import
		F1.4.2	Cleaning outliers
	RDP 3 Data construction	-	-
	RDP 4 Data integration	-	-
	RDP 5 Data formatting	F1.2	Changing parameters for import
		F1.4.1	Coding features
Modeling (RM)	RM 1 Model selection	F1.4	Managing features
	RM 2 Model preparation and tests	F3.1.1	Training a model
		F3.1.2	Balancing (training) classes
	RM 3 Model creation	F3	Managing models
		F3.1	Creating a model
		F3.2	Saving and downloading a model
		F3.3	Deleting a model
		F3.4	Uploading a model
		F3.5	Re-training a model
	RM 4 Model assessment	F4	Comparing models
		F6	Exporting results
Evaluation (REV)	REV 1 Evaluation of results	F6	Exporting results
		F5	Testing models
		F5.1	Checking a credit decision
		F5.2	Testing model performance with other datasets
		F3.2	Downloading a model
	REV 2 Evaluation of processes	F6	Exporting results
	REV 3 Determine the next steps	-	-

).

The second part of data understanding is data description (R_DU 2), which must include the functionalities to check “gross” or “surface” properties of the acquired dataset and to report on the results. Furthermore, the third part consists of data exploration (R_DU 3) to gain more insights for data preprocessing with the help of plots and reports (F₂ and F_2.1). Based on the nature of credit assessment, it is necessary to consider periodical changes, so the ability to compare multiple datasets must be provided to gain deeper insights for pre-processing datasets later in model training (F_2.2). The use of plots is ideal for supporting this task, as they provide a quick and intuitive understanding of the distribution of the data. Plots such as bar charts, correlation matrices, or boxplots are common types that are used to examine the datasets in data mining. The last step of data understanding is data assessment to verify the quality of the data by checking it for completeness, missing values, and errors (F_1.2 and F_1.3). No function is derived for the data assessment, as this check should be performed most easily by examining and making individual queries away from the artifact.

4.1.2. Data Preprocessing

The pre-processing consists of all the steps required to construct the final dataset. The tasks in this step are usually conducted more than once in a non-linear process. The first step consists of data selection (R_DP 1), which includes the functionality for the selection of features based on criteria such as relevancy, quality, quantity, and technical characteristics like data scope or type. Therefore, it is necessary to extend the functionality of F₁ to allow users to manage the features (F_1.4). The selected dataset must be clean of any missing values and outliers based on the conducted feature selection (F_1.2 and F_1.4.2).

Furthermore, the next part of data preprocessing consists of data construction (R_DP 3) to derive further metrics from a dataset using mathematical operations. However, the credit assessment dataset is ordinal, i.e., its values are subject to a sequence, and numbers are assigned to the different values. Therefore, the dataset does not require any conventions that are needed for other use cases dealing with more complex datasets, such as insolvency forecasting. Likewise, the integration of datasets, e.g., from multiple tables or records to create new records or values, is not required for the scope of the prototypical implementation of the AI dashboard for credit assessment. Moreover, the formatting of the dataset consists of syntactical modifications such as delimiter in CSV files or the position of the target variable (F_1.2). Therefore, it is necessary to have the option to change parameters for importing the dataset, so that the dataset can be imported properly. Furthermore, in the case of metrical data records, it is necessary to have the option of converting them into an ordinal structure (F_1.4.1), which extends the derived function F_1.4.

4.1.3. Modeling

The artifact must provide the structure for AI modeling, which includes the application of various techniques and the discovery of the (hyper-)parameterization leading to optimal values. It is important to consider that the performance of a model depends strongly on the dataset. Hence, modeling techniques require certain pre-processing measures, which is why it is essential to be able to go back to an upstream phase. The first step for modeling begins with the selection (R_MB 1) of modeling techniques, such as neural networks, decision trees, support vector classifier, XG Boost, etc. To evaluate the quality and validity of the model, it is necessary to develop a mechanism that tests the performance of the model (R_MB 2). For a credit assessment, it is common to determine the error rates of classification. Therefore, it is necessary to be able to split the dataset into training and test sets. Credit assessment datasets usually contain more approvals than rejections, which is why a balancing function must be offered to avoid any bias in the trained models.

The creation of a model (R_MB 3) consists of the determination of many parameters (F₃). Moreover, this step involves several iterations, including creating (F_3.1), saving (F_3.2), deleting (F_3.3), uploading (F_3.4), and re-training (F_3.5) an AI model. As a result, models can be evaluated based on the performance metrics. As the decision structure and therefore the datasets for credit assessment can change periodically, it is necessary to compare different datasets and models. For the assessment of the model (R_MB 4), many technical areas usually come together to evaluate the results. There is a large overlap between this step and the subsequent evaluation phase. First, the data mining engineer or data scientist interprets the results using their knowledge and the success criteria. In addition, business analysts and experts in their respective fields are consulted to discuss the results from the company’s perspective. In most projects, the data scientist/data mining engineer needs to create several models so that they can be compared with each other. Therefore, the functionality to compare the models and export the results is required to assess the models.

4.1.4. Evaluation

This step provides the functionality to evaluate the quality of the models and the approach, including data preparation and data quality. It is important to check whether the business-related problems of credit assessment, the proposed solution, and the AI model have been sufficiently considered. Previous assessment steps (R_M 4) deal with factors that affect the performance of the models, in which context the objective of this phase is to evaluate whether a model contributes to the achievement of the company’s objectives and whether it fulfills the business-related requirements (R_EV 1). It is therefore necessary to view (F₅) and export (F₆) the results. This makes it possible to check and compare all the specifications of the models. In addition, the models should be able to be tested with both real as well as hypothetical credit applications (F_5.1) or using another dataset (F_5.2) to simulate the model behavior. The function of downloading the model (F_3.2) is ultimately to integrate it into a production environment or to make further adjustments (R_EV 2).

The entire process up to model creation should be documented for evaluation, both to fulfill regulatory requirements and to ensure the traceability of the procedure. This requires that all model parameters and features can be exported in a report. In the final step of the evaluation phase, it then remains to be defined how to continue with the defined AI models. Hereafter, the development process is finished if the model performance is good enough and all requirements have been met, so that the deployment can begin. In addition, further iterations or new model development processes can be initiated if the model behavior needs to be tested regularly with new or revised datasets. However, although the artifact can help to make final decisions about the proposed AI models, it cannot provide any functionality to determine the next steps. An overview of identified requirements and derived functionalities is shown in Table 2.

4.2. Analysis of Use Cases of AI Dashboard for Credit Assessment

To structure the process based on the identified requirements and functions, it is necessary to analyze the use cases (Figure 3) and UML activity diagrams (Appendix C). This can promote the understanding of the requirements and functions when designing the artifact. Moreover, the individual modules do not correspond to the individual phases of the CRISP-DM process. The reasons for this relate to the iterative nature of CRISP-DM. For example, the first phase to be supported, data understanding, includes the steps of describing and examining the dataset. However, these steps cannot be completed before the dataset can be imported into the prototype correctly. Therefore, the function to change the parameters for import (F_1.2), which is assigned to the second phase (pre-processing) according to CRISP-DM, must have already been carried out previously.

The artifact’s functional scope is illustrated with a UML use-case diagram divided into four CRISP-DM phases (Figure 3), using colors (blue, yellow, green, purple) corresponding to four modules: upload, plot, model, and dashboard. The upload module assists with preprocessing and data understanding by allowing dataset uploads and formatting. The plot module enables graphical evaluation and detailed statistical analysis of datasets. The modeling module supports various modeling techniques and parameter optimization. The dashboard module visualizes and compares model performance, allowing new models to be uploaded or existing ones retrained. It also allows periodic testing of model behavior with new datasets or individual credit applications.

4.3. Selection of Modeling Techniques

The regulatory principles determine the criteria for the use of AI models in financial services. The European Commission regulation for the use of AI [10] stipulates that these models must have good forecasting capability and no systematic errors or distortions. In addition, regulatory principles for processes related to data quality as well as model validation are defined [21]. However, these are not considered in detail, as they must be implemented individually by the respective institutions for each specific application. For instance, they do not specify what can be considered good forecasting capability.

Moreover, the supervisory authorities define further country-specific regulations for the use of AI-based models. For example, the Federal Financial Supervisory Authority (BaFin) of Germany defines conditions for the selection of methods and techniques, requiring that companies must be able to justify the decisions and the underlying assumptions comprehensibly [20]. The use of models with increased complexity and so-called “black box” structure is critical for supervisory authorities, as they are concerned about the problem of limited insight into the model decision structure. Moreover, approaches such as LIME (local interpretable model-agnostic explanation) provide insights into the inner structures of AI models and so reveal the underlying decision structures of AI models [43]. However, the required explainability relates not only to the model but also to the associated process for its development, so documentation is essential to bring more transparency to the underlying development process. Therefore, the supervisory authorities provide no general approval for algorithms, so companies must ensure that the models produce correct, robust, and reproducible results.

The modeling techniques for credit assessment must provide a high level of explainability, transparency, and performance [10]. Nevertheless, for prototypical implementation, the modeling process must allow the use of different approaches that can be evaluated and compared. Therefore, the artifact should be expandable to integrate further modeling techniques. However, users should be aware that complex models such as neural networks cannot be used in credit assessment without further elaboration by supervisory authorities due to the hidden black box structure. We derived the given categorization and techniques, i.e., statistical and intelligent learning models and ensemble methods, from the literature [13,15,44,45,46,47,48] to provide algorithms that were as representative and efficient as possible (

Table 3. Selected modeling techniques for credit assessment dashboard.

Intelligent methods	Artificial neural network
	Decision tree
	Support vector classifier
	K-nearest neighbors classifier
Statistical models	Linear regression
	Logistic regression
Ensemble methods	XGB classifier
	Random forest

). However, a clear categorization of the selected techniques was not possible since semi-parametric methods represent a bridge between statistical and intelligent learning models.

To appropriately compare models, it is necessary to provide functionality for model assessment based on performance metrics. Graphs and tables can support this assessment process by visualizing extracted performance metrics [49,50,51]. A suitable representation of model performance is the confusion matrix, which visualizes performance using actual and predicted values, aiding in decision making [49]. Additionally, the receiver operating characteristic (ROC) curve effectively illustrates model performance by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. A steeper curve closer to the 0.0/1.0 coordinate indicates a better model, whereas a bisector-like ROC curve suggests random classification [52]. It is crucial to use statistical and appropriate metrics to present performance neutrally without distortion [13,53,54,55]. Chen et al. [13] define relevant performance metrics for binary classification or regression, such as in credit assessment. Moreover, including the negative prediction value (NPV) is essential for assessing the predictive quality of negative classifications, particularly in credit assessment, where an AI model could inaccurately achieve high accuracy by classifying all applications as positive if the dataset has few bad credit applications.

Moreover, it is essential to include these metrics as a part of the exported reports (

Table 4. Metrics for the assessment of the model performance of a binary classification.

Value	Name	Description	Calculation
Acc	Accuracy	-	${\displaystyle \frac{TP+TN}{n}}$
RSME	Root mean squared error	Equivalent to the Brier score for one-dimensional predictions	$\sqrt{{\displaystyle \frac{\sum {(P\left(x\right)-T(x\left)\right)}^2}{n}}}$
TPR	True positive rate	Recall, sensitivity, hit rate	${\displaystyle \frac{TP}{TP+FN}}$
TNR	True negative rate	Specificity, selectivity	${\displaystyle \frac{TN}{TN+FP}}$
FPR	False positive rate	Fall-out	${\displaystyle \frac{FP}{TN+FP}}$
FNR	False negative rate	Miss rate	${\displaystyle \frac{FN}{TP+FN}}$
PPV	Positive prediction value	Precision	${\displaystyle \frac{TP}{TP+FN}}$
F	F-score	Harmonic mean of precision and recall	$2\ast {\displaystyle \frac{PPV\ast TPR}{PPV+TPR}}$
AUC	The area under the ROC curve	-	-
NPV	Negative prediction value	-	${\displaystyle \frac{TN}{TN+FN}}$

) and to export training and testing datasets for further assessment, e.g., by responsible regulatory authorities or managers [13,51,56,57,58]. Documentation and export of all model-related information can promote development and evaluation processes that are as transparent and comprehensible as possible. Due to the required model assessment (R_M4), it is crucial to be able to compare these performance metrics of several models with drill-down functionality to identify and highlight their differences [50,59,60,61]. Thereby, the grid structure of a table can provide a simple, clear comparison of the different performance metrics. The possibility of adding different models to the table allows the user the necessary interactivity [62].

4.4. Design and Development

To fulfill the specified requirements to assist the process of AI development, we designed and developed the web-based artifact AIDash. Web-based applications enhance transparency and traceability in the development process by providing easy access to artifacts for operational employees, managers, and regulators [63]. They facilitate real-time collaboration, ensure accountability through detailed logs and version control, and improve communication via centralized information and interactive dashboards. However, these benefits come with challenges such as increased security risks, dependence on internet connectivity, complex implementation, and compliance with data privacy regulations. Balancing these advantages and disadvantages is essential for effectively utilizing web-based applications while mitigating potential risks. For the development of the artifact, we opted for the Python-based Django framework, which provides sufficient AI modules and a modern development architecture for data mining processes [64,65] (Appendix H).

As described in Section 4.2, the artifact consists of four modules (upload, plot, model, and dashboard), that guide the user through the entire model creation process (Figure 3). Below, a possible sequence of use is modeled using UML activity diagrams for each of the modules (Appendix C). However, as the procedure and the required processes are strongly dependent on the respective case and the stage of the project, completely different sequences are also possible. For instance, the data format check can be skipped if it has already been carried out manually.

4.4.1. Upload Module

The first module provides the functionalities to support the data understanding and preprocessing aspects of CRISP-DM. First, a new dataset can be uploaded in CSV format or selected from the list of already uploaded datasets (Figure 4). The artifact provides an overview of the uploaded or selected dataset so that the user can change the parameters for the import if there is incorrect formatting, e.g., caused by an incorrectly selected delimiter. Then, once the file has been read correctly, the target variable must be specified. The selected delimiter, the position of the header line, and the name of the target feature in the dataset are saved in the database.

Furthermore, it is possible to check, rename, or delete the columns if any changes are necessary for training an AI model (Appendix A and Appendix D). However, the management of the dataset (F₁) also provides functionality for coding the dataset (F_1.4.1), with the conversion of the metric characteristics into ordinal values such as credit ratings (AAA, AA+, …, D). In addition, there is another option for setting the value ranges for the assignment of ordinal values, which helps to exclude outliers in the selected columns by entering minimum and maximum values. For additional insights to be able to make necessary changes in import parameters, such as value ranges to exclude outliers or for an appropriate selection of features, the user can navigate to the plot module and generate graphical impressions with interactive bar charts and boxplots or can check the correlations between the features. Therefore, multiple iterations may be necessary to finalize the parameters and the selection of features.

4.4.2. Plot Module

The plot module serves as a graphical representation of the selected dataset (Figure 5). It thus enables in-depth analysis and comparison between, for example, the periodic datasets (e.g., D_t and D_t−1) to determine the possible conceptual or model changes. The potential deviations in datasets can lead to changes in the model structure, meaning that it is no longer possible to continue training existing models. Initially, the user decides whether only one dataset should be viewed or a comparison is desired. Depending on the decision, a second dataset from the user can be selected or removed from the view. The default view is a bar chart, so that the frequencies of each feature are displayed. The user can also select a pie chart to analyze the distribution of the feature values.

However, the primary focus on percentage composition (Appendix E) can be explored in more detail using the boxplot diagram, which provides a precise overview of the distribution, quantiles, and medians (Figure A9). Moreover, the boxplot diagram also shows outliers and provides an opportunity to remove them. As another important part of the data analysis, a correlation matrix can be selected to obtain the pairwise correlation coefficients of the features in the form of a table (Figure A11). AIDash allows us to make an easy comparison with a second dataset for each diagram type. Further arrangements of datasets based on the graphical analysis can be carried out in the upload module.

4.4.3. Model Module

The model module serves the management of models so that the user can create or upload new models and also delete or download existing models (Figure 6). The buttons for deleting or downloading are located on the left-hand side of the module. A new model can also be created using the corresponding button, whereupon a dialog window opens and allows the user to specify a title for the model. Once the desired model has been selected, individual hyperparameters for the model can be determined manually by the user (Appendix F). Furthermore, the ratio of “good” and “bad” credit applications in the selected dataset is plotted on a doughnut chart (Figure A12). Significant imbalances in the dataset can have a strong negative impact on model accuracy and distort the results by simply predicting the majority class and so providing poor recognition of bad credit applications [13].

Therefore, AIDash provides two options to balance the dataset by adjusting the ratio of “good” and “bad” credit applications (Figure A12). First, the random removal function allows the selection of the desired ratio between “good” and “bad” credit applications, so that credit applications are randomly selected and removed. As a second option, SMOTE–Tomek provides a combination of over- and undersampling techniques to balance the dataset [66]. Datasets for credit assessment contain a very small number of bad loans by default [67,68,69]. The SMOTE–Tomek algorithm randomly selects entries from the minority class and generates synthetic entries using the K-nearest neighbor algorithm until the desired ratio between the minority and majority class is reached [66]. Subsequently, so-called Tomek links are removed, which are defined as entries of the majority class whose nearest neighbor is an entry from the minority class and vice versa. In the implementation of the prototype, oversampling is carried out until a balanced ratio (50:50) between the majority and minority classes is reached. Once the desired adjustments have been made, the model can be trained (Figure A12).

In case of a need to evaluate or compare a model created independently (outside the prototype), it is possible to upload models (Figure 6). For this, a dialog window opens in which the title, description, model name, and model type can be specified. Furthermore, the associated training and test datasets must be uploaded. These must be serialized (e.g., using the Pickle library of Python) so that they can be imported and saved as a new object in the database. However, before the model is saved in the database, the performance metrics can be calculated. Once all relevant models are available, it is possible to navigate to the dashboard module for the evaluation of multiple models.

4.4.4. Dashboard Module

The dashboard module is the core of the prototype (Figure 7) and provides the necessary functionality to evaluate prepared or existing AI models (Figure 6). All previous steps and modules are used to build the model so that it can be evaluated in this module. Moreover, it also includes additional features, such as re-training, testing with another dataset, or creating a report (Appendix G). By default, the dashboard contains a single model including all key performance metrics for the evaluation. However, it is possible to add further models for comparison with the key performance metrics using the plus symbol in the top right corner of the table (Figure 7).

In addition, AIDash provides the option of downloading each model and the associated training/test sets via the corresponding button in the last row of the table. A full report can be created via the third icon, containing model performance based on metrics, metadata about the model, classifications made, training/test datasets, graphs, and the selected hyperparameters of the model (Figure A15). The dashboard module itself provides step-by-step documentation of the modeling process with the specific operations performed. Thus, documentation of the tests and analyses performed is provided either via the dashboard itself or by downloading reports for internal or regulatory approval.

Furthermore, each model contained in the view can be retrained via a drop-down menu by uploading new training and test datasets in serialized format (Figure A13). This function allows users to test existing models with new datasets to conduct further adjustments directly on the dashboard module. For this, a new model is created in the background so that new credit applications can be assessed. Moreover, the model performance can be tested either by entering credit applications manually via a drop-down menu or by file upload (Figure A14), so that the predictions for each entry can be calculated. Moreover, there is a binary prediction option in which the model classifies the application as “good” or “bad” (1—good; 0—bad). Alternatively, it is possible to generate and export the probabilities of the classification (between 0 and 1).

In case of a need to test already trained models with new datasets (e.g., for periodic comparisons between D_t and D_t−1) to determine whether the model performance is sufficient, or any adjustments are necessary, by selecting “Test model with another dataset” in the drop-down menu, the corresponding dialog window is opened (Appendix G) so that a title can be specified for the new object and a new corresponding testing dataset can be uploaded. Subsequently, the prototype generates the predictions and provides the performance metrics of the model.

4.5. Demonstration of AIDash

Following the implementation of the artifact, the functional correctness of the AIDash was validated by applying it to an application scenario. The demonstration of AIDash consisted of two consecutive periods and the datasets of these. Due to the necessary change of datasets, it was necessary to check whether there was data or model drift, so that a new AI model could be trained, tested, compared, and prepared for deployment.

Based on the periodic nature of credit scoring datasets, the first module (Section 4.4.1) provides the functionality to upload a new dataset to compare multiple datasets (between D_period1.csv and D_period2.csv) and model behavior. Moreover, after the final selection of features from the dataset (see Figure A5), the coding process of features can be conducted (see Figure A6). Since both datasets were already coded, it was not necessary to make further adjustments to the features, and the next step began with the comparison of the feature distributions between datasets, e.g., the current and the next period, on the plot module before training the model (Appendix A).

By using the comparison view, it was possible to identify possible data drifts that affected the model performance and the model behavior (Figure 8). Firstly, the ratio between “good” and “bad” credit applications was similar in the two datasets. However, the second feature (existing_account) showed that the third (for more than 200 monetary units or salary accounts for at least one year in the bank) and fourth types (for no current account) of customers were associated with an upward trend in credit applications compared with the previous period. In addition, we observed a new set of loan durations, with 10 loan applications between 54 and 60 months. In addition, an increased loan duration was also reflected in the amount of the loan applied for (9: 1500 < … ≤ 20,000). These significant changes indicated a data drift that may have led to incorrect assessment by the models.

Due to the identified changes in the new dataset, it was necessary to develop a new model to assess the performance and ensure system safety. To do this, the model module (Figure 6) provided the functionality to manage the modeling process. The definition of a new model with AIDash is assisted by the predefined parameters, which can be further refined if necessary. However, the new dataset contained more “good” than “bad” loans, which was balanced out during the development of the new models (Figure A12). To enable a comparison on the dashboard module to find the best among all existing modeling techniques, we created several models from the list of techniques provided by AIDash.

In the last step of the process, AIDash provides a comparison view of multiple models. In order to demonstrate the model evaluation process, XGB, and DT models trained with D_period2.csv and D_period1.csv were added so that the performance of each model could be analyzed (Figure 9). Thus, AIDash was able to promote a periodical comparison between the existing and newly trained models. However, in this step, it was necessary to iteratively add/remove models to/from this view for comparison before the final decision was made. In this case, the XGB and DT models allowed us to assess the transition in model performance between the two periods and datasets.

First, the ROC curve indicated that the XGB models overperformed the DT models from this and the previous period. We observed that the curve for the new XGB model with D_period2.csv rapidly extended towards the upper right-hand corner of the graph. Second, the new XGB model with the performance metrics in the table below showed that a change or modification of the existing model made sense due to its superior performance. In addition, the performance metrics showed the better quality of the XGB model that was trained with the new dataset, with higher accuracy, higher F1 score, and lower Brier score. The metrics are highlighted with specific colors to make it easy to identify changes in performance. Moreover, AIDash provides the functionality to test the model with another dataset, allowing monitoring of the performance of existing models on new datasets to identify model drift (Figure A13). Lastly, it was necessary to export the model’s train and test datasets and the documentation report to make final adjustments for deployment. In case of a need for any regulatory assessment, the model behavior can be assessed by checking individual credit applications (Figure A13).

5. Discussion and Limitations

The implemented artifact provides an exemplary structure based on the gathered requirements from CRISP-DM to develop, maintain, evaluate, and monitor AI models for financial services.

Table 5. Overview of the functional scope of the developed artifact based on CRISP-DM.

Steps	Business Understanding	Data Understanding	Data Preparation	Modeling	Evaluation	Deployment
1.	Determine business objectives	Collect initial data	Select data	Select modeling techniques	Evaluate results	Plan deployment
2.	Assess situation	Describe data	Clean data	Generate test design	Review process	Plan monitoring and maintenance
3.	Determine data mining goals	Explore data	Construct data	Build model	Determine next steps	Produce final report
4.	Produce project plan	Verify data quality	Integrate data	Assess model		Review project
5.			Format data

Green: full coverage; yellow: partial coverage; orange: No coverage.

provides an overview of the functional scope of the artifact AIDash. Comparing the scope of requirements and functions (Section 4.1) with the steps from CRISP-DM, the feasibility is visualized as follows: the green cells in the table are supported steps, while the yellow cells indicate the partial coverage, and the orange cells are the unsupported steps. The business understanding and deployment phases cannot be supported with the artifact as they require more strategic, conceptional thinking, as well as practical action and integration beyond the artifact. Assuming that the fundamental and strategic requirements are in place, the steps marked in green and yellow can be facilitated and improved within the scope of the artifact.

As in the first research question addressed, the derived functionalities from CRISP-DM outline the functional requirements in Table 2 [F₁–F₆]. In addition, these requirements can help to develop an artifact for other similar use cases working with quantitative datasets in financial services. Moreover, due to the regulatory requirements for the use of AI in financial services, it is necessary to extend the artifact with further explainability, traceability, and transparency approaches throughout the development process—from data preparation to model development. The regulations do not clearly define any criteria for a suitable level of explainability. The artifact provides decision tree and logistic regression models that include explainability by design (Section 4.3). However, for black box models such as NN, it is necessary to extend the artifact with XAI techniques such as LIME or Shapley to provide the required explainability for these models.

The existing research gap in appropriate structures and systems for the development of AI models including training, testing, comparison, evaluation, deployment, and documentation mechanisms was investigated in this study. The requirements identified from the CRISP-DM stages addressed the problem of long development times and concept shifts. The structure of the artifact was formed from the functions derived from the requirements and mapped into four modules (upload, plot, model, and dashboard). The AI models identified in Section 4.3 and legal requirements confirm that models cannot be permitted or prohibited in general. Therefore, dashboards must provide a structure that ensures that the models and the associated processes are transparent, explainable, performant, and individually assessable.

The problem of the occurrence of concept shifts and the associated decline in model performance can impair a model’s ability to classify credit applications based on factual characteristics. Corresponding models are often based on datasets from up to three years in the past, as the development of new models by data analysts is time-consuming [15]. This high level of effort comes from the fact that there are high quality requirements for these models and the documentation both from credit institutions and the regulators, in order to increase transparency and ensure customers’ safety and trust. Therefore, the adaptability of AI-based services must be ensured due to the dynamic macroeconomic environment, so that possible concept shifts can be recognized at an early stage and measures can be taken to ensure good predictive capability of the models (Section 5).

Furthermore, the performance of a model depends significantly on the dataset used, so multiple iterations are relevant for the preparation of datasets and models for determining the best set of configurations [13,70]. Therefore, the possibility to analyze, compare, test, and evaluate datasets and models in detail is provided in the four modules. Furthermore, the export function makes it possible to document systemically the reports on conducted analyses. A potential conceptual shift can easily be detected by testing the already trained models with datasets from a different period. A reduction or improvement in model performance can then easily be discovered using the calculated performance metrics. If further assessments are required, the user can create or upload new models directly.

All these factors lead to lengthy model development and therefore cost-intensive processes that are usually carried out by experts in the field of data science, which is exactly the problem addressed by the developed artifact AIDash. The four outlined modules (upload, plot, model, and dashboard) enable the rapid development or maintenance of AI models. Thanks to the graphical user interface, even non-experts in the field of data science can develop or evaluate AI models. However, it should be noted that these models are not considered final or optimal. Rather, they can serve as a first iteration, as a performance comparison, or for control of the performance of existing or used models.

Moreover, the plot module provides the functionalities to identify deviations from the previous datasets, based on different graphical representations. The function for comparing data records enables assessment and observation of features to determine whether any strong variations or imbalances exist. The corresponding graphical representations from the available datasets help to rapidly identify periodical changes in the datasets that may lead to a conceptual shift. Furthermore, proposed models must be configured on the model module, which provides the functionality to address the imbalances between “good” and “bad” loans in the selected dataset. In addition, the artifact provides the basic functionality for creating models and balancing the selected dataset, which can be extended in fine granularity depending on the objective and required complexity.

The AI dashboard for financial services has significant implications for regulatory authorities, especially under frameworks like the European AI Act. While it enhances transparency and traceability by documenting every step from data preparation to model development, the explainability requirements of the models need to be further investigated based on the specific modeling technique. Therefore, requirements for a model- and case-specific degree of explainability must first be investigated. Furthermore, regulatory authorities demand high levels of transparency and traceability, and the AI dashboard aids compliance by providing detailed documentation of all actions taken during the model development process, supporting regulatory reviews and helping maintain compliance with documentation standards.

Additionally, the dashboard’s functionalities to detect and address conceptual shifts are crucial for maintaining model performance, which is essential for compliance. Systematically analyzing and testing models with new datasets helps to identify shifts early, allowing timely corrective actions. This proactive approach ensures AI models remain accurate and reliable, aligning with regulatory expectations for ongoing validation and performance monitoring. Furthermore, the dashboard promotes standardization in AI model development, ensuring all processes are consistent and well-documented, which is essential for regulatory approval. This streamlined development process reduces time and costs, making AI model development more accessible, aligning with regulatory goals of accessibility and inclusivity, and enabling even non-experts to develop and evaluate models.

Moreover, documenting data preprocessing, modeling, and sampling enhances transparency for regulatory authorities. Sharing prepared models with all documentation and datasets will aid regulatory control and review. This documentation justifies decisions to use or change AI models. It lacks features for recalculating or combining new model features, which could improve prediction quality. Research indicates that splitting datasets to train multiple models can enhance predictions, suggesting that the artifact could benefit from this functionality to reduce bias and improve accuracy [19,71]. Lastly, the problem-centered DSR process should include practical evaluations and expert interviews to refine requirements, concepts, and prototypes based on practical insights [23].

6. Conclusions

This study investigates the development of an AI dashboard to structure and support the development process of AI models for private credit assessment. We identified 13 requirements as part of the first research question and derived 28 key functionalities from them, distributed across four blocks of CRISP-DM (Table 5): data understanding, data preprocessing, modeling, and evaluation. Due to the underlying dataset and corresponding use case of credit assessment, four identified requirements [R_DU 1, R_DP 3, R_DP 4, and R_EV 3] were not considered for deriving any functionalities. To provide a structured process of modeling, the proposed artifact consists of four modules equipped with the identified functionalities. To address the second research question, we structured the model development process and implemented an artifact that enables multiple iterations between these modules, allowing numerous corrections for models and datasets. Moreover, the provided workflow defines a useful and structured method for data preprocessing to avoid any bias in datasets. Automated documentation makes it possible to track the design process and thus brings more transparency into the field.

The AI dashboard for financial services significantly impacts financial regulations, particularly under frameworks like the European AI Act. It enhances transparency and traceability by meticulously documenting each step of the model development process, aiding regulatory reviews and compliance. This ensures all actions during data preparation, model training, testing, and evaluation are transparent and traceable, essential for regulatory scrutiny. The dashboard’s ability to detect and address concept shifts maintains model performance, aligning with regulatory requirements for ongoing validation and performance monitoring. It promotes standardization in AI model development, ensuring consistent and well-documented processes crucial for regulatory approval. This streamlines development reduces time and costs and supports regulatory goals of accessibility and inclusivity, enabling even non-experts to develop and evaluate models. While the dashboard addresses many current regulatory requirements, further enhancements are needed to fully comply with evolving European AI Act standards, to make it a comprehensive tool for regulatory compliance in AI in financial services.

The artifact AIDash goes beyond the functionalities of a typical dashboard and addresses several problems of model creation and validation in credit assessment. In particular, the prototype addresses the problem of concept shifts (Section 5). Users can recognize a decrease in model performance by testing an existing model with a new dataset, e.g., from a subsequent period. Furthermore, thanks to the graphical presentation (plot module), changes in the dataset can be recognized and countermeasures can thus be undertaken. The dashboard as a web application enables simplified analysis and evaluation of datasets and models so that even non-experts in the field of data science can perform regular analyses and monitor already developed models.

Extensive support for the phases and steps of CRISP-DM enables initial models to be developed quickly and easily. These models can be used, for example, as a performance comparison or can also be further developed so that they can subsequently be deployed in a productive environment. The AI dashboard stores all data relevant for the documentation of model creation. These data can be exported and thus also stored externally so that they can be used for the mandatory documentation for the supervisory authorities. In addition, the explainability desired by regulators must be addressed so that the artifact can be extended by XAI techniques to allow more complex models such as neural networks to be used in financial services. This study has made it clear that further research is needed to define compliant approaches to design, develop, maintain, document, and deploy AI models.