Agile Development Methodologies and Natural Language Processing: A Mapping Review

Abstract

Agile software development is one of the most important development paradigms these days. However, there are still some challenges to consider to reduce problems during the documentation process. Some assistive methods have been created to support developers in their documentation activities. In this regard, Natural Language Processing (NLP) can be used to create various related tools (such as assistants) to help with the documentation process. This paper presents the current state-of-the-art NLP techniques used in the agile development documentation process. A mapping review was done to complete the objective, the search strategy is used to obtain relevant studies from ScienceDirect, IEEE Xplore, ACM Digital Library, SpringerLink, and Willey. The search results after inclusion and exclusion criteria application left 47 relevant papers identified. These papers were analyzed to obtain the most used NLP techniques and NLP toolkits. The toolkits were also classified by the kind of techniques that are available in each of them. In addition, the behavior of the research area over time was analyzed using the relevant paper found by year. We found that performance measuring methods are not standardized, and, in consequence, the works are not easily comparable. In general, the number of related works and its distribution per year shows a growing trend of the works related to this topic in recent years; this indicates that the adoption of NLP techniques to improve agile methodologies is increasing.

1. Introduction

Agile software development has boomed in the last two decades, being one of today’s most important software paradigms. Agile development aims to enable an organization to be faster and flexible [1]; one of the most important bases of agile development is the continuous delivery of software as established in the agile manifesto principles [2]. These principles imply that different methodologies have common bases, such as the iterative approach of small deliveries of functional software. One of the principles establishes that “the software working is the main measure of progress” [2]; constant interaction with clients is another fundamental principle maintained throughout the development.

Software development teams currently use agile methodologies depending on what they want to prioritize. For example, Scrum [3] is an agile methodology that reduces the documentation artifacts compared to traditional development methodologies. Iconix [4] is a semi-agile methodology that also reduces artifacts, but retains some important ones, including the robustness diagram [5]; eXtreme Programming (XP) [6] is intended to improve software quality and responsiveness to changing customer requirements; and Crystal [7] is considered as a lightweight or agile methodology that focuses on individuals and the interactions.

According to the agile manifesto, the best architectures, requirements, and designs emerge from self-organizing teams [2]. Self-organization of teams is fundamental to carrying out the documentation process in an agile project to accelerate that process. One of the main points is that self-organization implies no rigid standards for accepting documentation, which speeds up the documentation process. However, this documentation flexibility also entails problems such as inadequate documentation of requirements if the team organization fails [8].

The lack of adequate documentation of changes during an agile project leads to a problem called technical debt. This debt is generated by the activities of the development team members when they do not do certain project activities in due time, preventing or hindering future development [9]. The longer one waits to carry out the postponed tasks, the more costly (in terms of time) it will be for the project, and the more difficulties will carry with it.

It is common to receive changes throughout the project’s development; in fact, accepting changes is established in a principle of the agile manifesto [2] to provide a competitive advantage to customers by development flexibility. This situation means that documentation must be updated constantly throughout development. However, developers do not update documentation for different reasons such as lack of time caused by high priority delivery [10] or reluctance to documentation [11].

Some methods to reduce technical debt and documentation problems are based on simplifying the requirements gathering [12]. Some others rely on text analysis to obtain information from documentation, such as code comments, to find information about the project’s functionalities [13]. A specific method that could be used to develop text analysis tools is natural language processing (NLP) [14], which belongs to the Deep Learning [15] field. NLP consists of a range of theoretically motivated computational techniques to analyze and represent texts to achieve human-like language processing. It can be divided into three broad categories: lexical analysis, semantic analysis, and discourse analysis. An NLP system that contemplates the three categories is called a complete NLP system. This type of system should perform complex tasks such as paraphrasing input texts, translating a text from a source language to a target language, responding to questions about the content of a text, and producing inferences from an input text [15]. Various techniques and toolkits have already been developed and tested in the NLP area that quickly and efficiently works on text analysis. Stanford CoreNLP [16] is a reasonably robust tool that allows you to use various NLP techniques with a single tool. The Natural Language Toolkit (NLTK) [17] is another tool with a wide variety of modules, such as parsing, tagging, and classification, among others. New toolkits are recently under development, as with Stanza [18], a Python-based tool supporting 66 languages. It has many techniques similar to the previous ones, but with some new modules that include newer NLP techniques.

NLP has been used to process requirements, gathering artifacts such as use cases [19] and user stories [20]; in fact, there is a preference for using NLP to process user stories [21]. This preference may be due to the user stories’ characteristics [22], i.e., simple texts which are structured so that the most important elements of each requirement can be identified and their quality can be checked [23]. User stories are a method of representing requirements using a simple structure such as “Who?”, “What?”, and “for what?” [24]. In agile development, it is common to use user stories as an artifact for requirements gathering [21]. User stories allow the users to express requirements simply; this helps to improve the requirements gathering process and makes the process faster [23]. A quality standard is established to write user stories; that is, the quality of what is expressed is controlled to avoid ambiguity and problems that may lead to a misunderstanding of the requirements [25].

Agile requirements engineering (ARE) is a flexible and fast requirement gathering process that uses user stories as artifacts to collect requirements [26]; this process is iterative and incremental. Unlike traditional requirements engineering, where requirements gathering takes place only in the early phases of the project, agile requirements engineering allows clients to interact and communicate with the development team throughout the entire project development cycle [27,28]. According to [29], software requirements are written in natural language to improve communication with other members of the development teams in such a way that requirements are understandable for all the stakeholders [30]. The capacity of NLP for data processing leads to exploring solutions to requirements-related problems [31].

This paper presents a Systematic Mapping Review (SMR) to show the existing research on applying NLP techniques and toolkits in the documentation process of agile development methodologies. The mapping was carried out following the method proposed by Petersen et al. [32]. The rest of the document is structured as follows. Section 2 presents background concepts about NLP and related works. Section 3 presents the materials and methods including the mapping planning, research questions, search strategy, and selection strategy. Section 4 presents the results found in the mapping. A discussion of findings is presented in Section 5. Finally, in Section 6, the conclusions of the work are presented. The articles considered relevant for this mapping are presented in Appendix A.

2. Background and Related Work

NLP is the ability of computer programs to understand human language as it is spoken and written [15]. NLP’s more important applications include such as (1) machine translation (the learner must read a sentence in one human language and emit an equivalent sentence in another human language), (2) parsing (the process of determining the syntactic structure of a text), and (3) text classification (categorization of the elements of a sentence as nouns, verbs, adjectives, and adverbs) [14,15].

Different NLP techniques enable researchers to solve problems using one or many mixed techniques. Additionally, there exist NLP toolkits [33] that bring researchers a variety of NLP techniques in a simple way to use and implement, which eases the development of NLP systems. On the other hand, various evaluation methods have been established for measuring the performance of an NLP system [34].

To better understand the different NLP techniques, we describe some of the most common below, grouped by categories based on their purpose.

• Processing Techniques: This category encompasses those techniques that process the information of the text to find and extract specific data to be analyzed.

  –

  Part-of-Speech (POS) Tagging. This technique categorizes the elements of a sentence as nouns, verbs, adjectives, and adverbs. The labeling grants that differentiation of the elements of a sentence or a complete text inclusive [35].

  –

  Tokenization. The tokenization process consists of converting some text into a correspondent representative token; this token is an object that could be an array, an alphanumeric string, a number, or a customized object. Tokenizing text elements can be from a word-to-word conversion or a group conversion of sentences that share characteristics of interest. Before applying another technique, this technique is commonly used as a pre-processing or first step in NLP pipelines. Even POS Tagging has an implicit tokenization step to carry out the tagging [33].

  –

  Parsing. It consists of identifying the syntactic structure of a text by analyzing its words based on the language’s grammar of the given text. One important element of this technique is the parsing tree, which represents the structure of the text and can be represented visually [36,37].

  –

  Chunking. This technique can be defined as a process used to identify parts of speech and even short phrases present in a sentence. The identification goes hand in hand with grouping these for later analysis by identifying all the nouns, verbs, adverbs, etc. [38].

  –

  Lemmatization. The goal of this technique is to reduce the inflectional and derived forms of a word to a common base form for the entire text. A word’s base or simple form is a lemma, hence the technique’s name. Unlike a simpler technique called stemming, which heuristically clips a word in the hope that clipping will obtain the base form of that word, lemmatization is carried out through morphological analysis and the use of a vocabulary dictionary for proper and efficient conversion [39].

• Text representation: This category encompasses those techniques that transform the input text to obtain a new representation of the same information on a different format.

  –

  Word2Vec. This technique focuses on word-centered processing, which can be approached as a bag of words to see the importance of words in a text, the frequency and weight that each one implies, and to seek to perform certain operations to obtain information as the main topic of a text [40].

  –

  Bag-of-Words. The bag-of-words (BoW) model is one of the simplest feature extraction techniques used in many natural language processing applications, such as text classification, sentiment analysis, and topic modeling. Bag-of-words is built by counting the number of occurrences of unique features such as words and symbols in a document. The bag-of-words model is a representation that turns arbitrary text into fixed-length vectors by counting how many times each word appears. This process is often referred to as vectorization [41].

• Extraction techniques: This category encompasses those techniques that process the input to extract desired data. The input could be text or the output of processing or text representation techniques.

  –

  Semantic Analysis. Semantic analysis is the process of understanding a text written in natural language based on context and meaning. In general, semantic analysis processes the whole text to identify the real meaning of a text using the identification of elements and to assign logical and grammar roles to each based on the complete analysis [42].

  –

  Correlation and Dependencies Analysis. A dependency analysis and a correlation analysis are closely related to parsing. Both seek to identify the elements that make up the text structure and the relationships between them, and their impact on the functionality of the wanted text message. This process can be seen as relevant to POS Tagging [43].

  –

  Sequence Analysis. This technique is based on the use of number sequencing generated by tokenizing elements of a text corpus. By assigning a number to each word present in the text, different sequences found in the text can be generated and analyzed and, in this way, locate the importance of the order of the words or the effect it has on the text when analyzing other elements of it [44].

• Grouping techniques: This category encompasses those techniques that process the input to create groups based on some specified characteristic of the data.

  –

  Clustering. It is based on locating different words or sentences and grouping them by some common characteristic; the greater the similarity of the text, the closer they will be. The topics of different sentences could define similarity, the type of language used, or even the text grouping depending on which person is referred to by each sentence. The purpose of clustering is to create a dynamic number of groups (compound of similar words or sentences) without the need to analyze all the text manually [45].

• Neural Networks. Although artificial neural networks are not an NLP technique, it was decided to group all those approaches that use different types of neural networks in this category. Some types are recurrent networks (RNN), convolutional networks (CNN), or Long Short Term Memory (LSTM). This type of approach uses neural networks for the processing of text data, either for classification or obtaining information [15].

Related Work

To our knowledge, no secondary studies focused on using NLP to improve processes within agile development methodologies. However, some studies address similar issues, focusing on specific aspects, such as analyzing user stories using NLP [46]. Other studies focus on the general use of intelligent tools in agile methodologies [47]. Wagner et al. [48] discussed the importance and the potential advantages of documentation written in natural language in agile methodologies.

The above-cited works show NLP techniques used in various ways to work directly with user stories [46]. Additionally, they show the challenges encountered in applying NLP techniques, such as the need to improve validations of performance during the training of NLP applications. Another challenge is to find adequate data sets for testing and training; these data sets must contain enough data to ensure the efficient learning of the tool during its training process. Further, the cited works identified a challenge in creating tools independent of the context so that they can be generalized to new data that was not presented in the training phase, such as incorporating human intervention in performance evaluation to complement the effectiveness of the tools [15,34].

3. Materials and Methods

The method was based on the guidelines proposed by Petersen et al. [32] to carry out systematic mapping reviews. The method includes objectives, research questions, search strategy, and selection criteria. The study focuses on finding the applications of NLP in agile methodologies to identify, characterize, and summarize the research of the evidence in the literature on the use of NLP focused on improving the documentation process of agile development methodologies.

3.1. Definition of Research Questions

The main question that we want to answer with this review is the following:

What approaches based on natural language processing techniques exist in the literature focused on improving the documentation process of agile development methodologies?

Due to the main question being too broad, we divided it into four more specific questions, as shown below.

• RQ1. What approaches have been made to improve the agile documentation process using NLP techniques?
• RQ2. What NLP techniques and toolkits have been used to improve the documentation process of agile development?
• RQ3. How is the performance measurement of the NLP techniques when applied to support the documentation process of an agile project?

3.2. Search Strategy

Databases selection was made based on the access to them as follows:

• ACM Digital Library;
• ELSEVIER—ScienceDirect;
• IEEE Digital Library;
• Springer;
• Wiley.

We selected general terms taking into account the general topics of the research:

• NLP;

  –

  Natural Language Processing;

  –

  Natural Language;

  –

  NLP.

• Agile Methodologies;

  –

  User Story;

  –

  User Stories;

  –

  Agile;

  –

  Documentation.

Depending on the database, a general search string was defined and adapted to each search engine. Below, we present the search string.

(“Natural Language Processing” OR “Natural Language” OR NLP) AND (“User Story” OR “User Stories”) AND (Agile OR Documentation).

3.3. Selection Strategy

Inclusion and exclusion criteria were established according to Petersen guidelines [49]. The inclusion criteria were defined as follows.

• Primary studies published in journals and conference proceedings
• Studies in which the main topics were NLP, Agile methodologies, and agile documentation;
• Studies published in the last ten years to consider the recent research works in the field.

Now, the exclusion criteria were the following.

• Articles which not published in English;
• Secondary or tertiary studies.

Quality Assessment

An essential part of the selection strategy is the quality assurance of the papers. The quality assessment was made along with the full text review, see Figure 1 for details. We considered the results presented of each paper to decide whether an article has an acceptable quality level.

An article was considered of low quality when it had two or more of the following characteristics.

• It presents results that do not correspond to the information expressed in the paper.
• It presents results without any performance measurement.
• The paper does not present the information necessary to replicate the work.
• The paper results are based on assumptions or decisions that could be biased.

3.4. Conducting the Review

We planned this review between July and September of 2021; we executed the plan in two search periods. The first began in October of the same year, and the second in May of 2022. The selection process was carried out in partnership with a second researcher and the author to ensure the veracity and consistency of the process. The review process results were similar, but with low variations in the number of papers found due to the time process. We carried out the original search in October 2021; the pair review was carried out in November 2021, resulting in two more articles found on average in each database. The original and pair search was carried out for the second search in May 2022 with the same results.

Study Selection

Figure 1 shows the article selection process. We conducted the first search process using the search string and databases defined in Section 3.2. We found 617 articles in this first search; then, we selected 165 papers in the title review; as a result of the abstract review, we selected 85 papers; and finally, we accepted 56 papers when we reviewed the full-text of the papers, which includes the quality assessment process. For example, we discarded some papers because they did not present enough data to replicate their work and did not present their results. Other papers were discarded because they did not present any performance measurement, and the information presented was confusing or did not correspond with their results. We applied the inclusion and exclusion criteria among all the studies’ selection processes.

4. Results

In general, we analyzed the full texts of 56 articles. We present the results below organized by research questions.

4.1. Summary of Studies

Below, we present a summary of the accepted papers to explain the characteristics of the complete set of works that we analyzed to answer the research questions.

A total of 56 relevant articles were identified from the review, of which 17 (30%) are journal publications, while 39 (70%) are conference publications. The relevant studies were distributed in different databases, indicating no predominant database where authors prefer to publish on the subject. Around half of the articles analyzed are preliminary studies (27 articles). The other 25 articles present at least one proof of concept of the expressed idea, while 26 presented at least laboratory experiments with their datasets or were based on real industry open source projects. None were applied in a real-world environment or tested in a company environment.

Figure 2 shows the distribution of the relevant articles from 2010 to 2022. It can be noted that the years with the most relevant articles are 2020 and 2019. In addition, a clear upward trend can be observed until 2020, before falling abruptly in 2021. However, this behavior could change in 2022, when almost the same papers are published. There is also a more significant number of articles published in conferences compared to those published in journals.

4.2. (RQ1) What Approaches Have Been Made to Improve the Agile Documentation Process Using NLP?

During the analysis of complete texts, we found that a few works (three papers) focused on working on a specific agile methodology: two focused on Scrum and one on Extreme Programming. On the contrary, it is preferred to focus on common problems without distinguishing which method is performed when presented. The use of user stories as a fundamental artifact was found in 27 papers; other artifacts used in the selected papers are use case diagrams, use case scenarios, goal models, and class diagrams. Additionally, the findings suggest that it is preferred to focus on documentation artifacts used in various methodologies.

We identified four main categories in which NLP is applied in the agile documentation process:

• Improve Requirements Engineering: This category is about improving all the requirements engineering that includes recollection, analysis, validation, and management of requirements. Some works are talking about being a tool for the recollection of requirements [P19, P24, P43], these approaches focus on making more accessible the recollection process. Some approaches in this sense intend to improve the recollection using NLP tools to analyze the clients’ requirements on audio or text to produce a clear requirement or artifact with the information. Other approaches focus on enhancing the quality of the requirements on the artifacts [P5, P6, P7, P47]; some papers did this by verifying the quality of user stories and detecting redaction problems like redundancy on other artifacts.
• Artifacts Transformation: This category is about transforming an artifact, such as a user story, into another artifact corresponding with the same information, such as a use case scenario. The approaches center their efforts on specific artifact conversion, taking the original as the basis of the process. The transformations that we found in the relevant papers include the following: user stories to use case diagrams [P15], user stories to use case scenario [P10, P33], user stories to conceptual models [P8, P28, P54], use cases to class model [P2], user stories to BPM processes [P32], user stories to goal model [P30], user stories to class diagram [P40], and user stories to behavioral UML models [P50]. The methods used to transform artifacts are implemented using distinct NLP techniques.
• Agile Software Development Process: The papers grouped in this category intend to improve one or many of the elements of the agile development process. The generation of artifacts like estimation of time and cost [P13, P51], acceptance tests [P1]; and code generation of unit tests [P44]. This category includes works focused on project management [P22], prototype generation [P23] and product release [P17].
• Team Communication: This category focuses on improving team members’ communication. Some approaches focus on processing information with artificial intelligence (AI) to improve the project information and share it with the team [P31, P41]. Others focus on designing a domain language to facilitate communication among the team members [P39, P51].

Table 1. NLP application categories.

Category	Freq.	Studies
Improve Requirements Engineering	24	[P3], [P4], [P5], [P6], [P7], [P11], [P16], [P18], [P19], [P20], [P24], [P25], [P27], [P29], [P36], [P37], [P43], [P46], [P47], [P48], [P49], [P52], [P55], [P56]
Transformation of Artifacts	11	[P2], [P8], [P10], [P15], [P28], [P30], [P32], [P33], [P40], [P50], [P54]
Agile Software Development Process	15	[P1], [P9], [P12], [P13], [P17], [P21], [P22], [P23], [P26], [P34], [P38], [P42], [P44], [P45], [P53]
Team Communication	6	[P14], [P31], [P35], [P39], [P41], [P51]

shows the classification results by categories of the applications shown in the relevant papers. The most common category found was improving requirements engineering with 24 papers. Figure 3 shows the distribution of the frequency of papers found on each category by year.

4.3. (RQ2) What NLP Techniques and Toolkits Have Been Used to Improve Any Agile Methodology?

To answer question RQ2 in more detail, we divided the findings into two categories: NLP Techniques and NLP Tools.

4.3.1. NLP Techniques

The most used NLP techniques were classified as shown in

Table 2. NLP techniques applied in studies.

Category	NLP Techniques	Freq.	Studies
Processing Techniques	POS Tagging	23	[P2], [P3], [P5], [P7], [P10], [P11], [P15], [P18], [P19], [P27], [P28], [P29], [P31], [P33], [P37], [P39], [P40], [P42], [P44], [P46], [P47], [P50], [P52]
	Tokenizer	13	[P2], [P5], [P6], [P11], [P14], [P19], [P27], [P39], [P42], [P44], [P46], [P50], [P52]
	Parsing	11	[P2], [P5], [P6], [P7], [P10], [P11], [P13], [P14], [P30], [P34], [P50]
	Lemmatization	4	[P7], [P19], [P27], [P50]
Text Representation	Word2Vec	7	[P21], [P36], [P37], [P42], [P44], [P45], [P53]
Extraction techniques	Semantic Analysis	5	[P16], [P25], [P41], [P47]
	Correlation and Dependencies Analysis	6	[P10], [P27], [P32], [P33], [P50], [P52]
	Sequence Analysis	4	[P4], [P20], [P24], [P55]
Grouping techniques	Clustering	3	[P4], [P24], [P25]
Uncategorized	Neural Networks	3	[P22], [P36], [P37]
	Other	9	[P7], [P14], [P17], [P35], [P48], [P49], [P51], [P54], [P56]
	Not Defined	10	[P1], [P8], [P9], [P12], [P23], [P26], [P35], [P38], [P41], [P43]

. Note that each article can present more than one technique and could be counted in many categories. Part of Speech Tagging (POS) is the most used technique with 23 papers. After that, eleven papers used the Tokenizer technique. Additionally, Parsing techniques are used in ten papers. Six papers use Word Analysis like Bag-of-Words, Keyword extraction, and Word embedding. The rest of the techniques are Semantic Analysis (5 papers), Correlation and Dependencies analysis (6 papers), Sequence analysis (4 papers), Clustering (3 papers), Lemmatization (4 papers), and Others (9 papers). Finally, the Not Defined category includes ten articles that did not specify the applied technique or articles that proposed some techniques but they were not implemented. In Figure 4, the frequency distribution with which the different NLP techniques are used through the years can be observed. It can be seen that POS Tagging is the technique that he has mastered for the last three years.

4.3.2. NLP Tools

The most used tools presented in the selected articles are Stanford CoreNLP, with five papers, followed by Natural Language Toolkit (NLTK), with three papers. SpaCy was used in two papers, and Stanford POS Tagger with one paper. Additionally, some papers show their own tools derived from existing ones or completely new tools such as Cortical.io, IdeaSy, and Gherking20AS. Although some new tools were found, there were authors’ approaches and need more tests to be considered and published as a toolkit. Figure 5 shows the frequency of the toolkits found in the reviewed papers. A brief description of these main tools is presented below.

Most of the tools have the POS Tagging function, which is the most used in the selected articles, as seen in Figure 3. The usefulness of this technique goes hand in hand with the chunking technique (see Section 2 for background concepts). Through the fragmentation of a phrase, it is possible to obtain smaller elements that can be labeled to identify to which part of a certain sentence it belongs.

Table 3. NLP tools.

NLP Tool	Features	Freq.	Studies
Stanford CoreNLP	Tokenization, Part-Of-Speech (POS) Tagging, Lemmatization	5	[P6], [P9], [P11], [P40], [P42]
Natural Language Toolkit (NLTK)	Tokenization, Part-Of-Speech (POS) Tagging, Dependency Parsing, Lemmatization	3	[P5], [P6], [P35]
SpaCy NLP	Tokenization, Part-Of-Speech (POS) Tagging, Dependency Parsing, Lemmatization, Similarity Analysis	3	[P10], [P18], [P52]
Stanford POS Tagger	Part-Of-Speech (POS) Tagging	1	[P22]

shows the principal tools found in the selected studies.

• Stanford CoreNLP [50]: This is a complete tool to perform different operations such as tokenization, POS Tagging, and Lemmatization. It is a toolkit written in Java that can be used freely, having data dictionaries available for various languages such as English, Chinese, Spanish, French, German, Italian, and Arabic.
• Natural Language Toolkit (NLTK) [51]: NLTK is a platform written in Python that allows it to work easily and quickly with language processing. It includes various elements and libraries that allow the user to carry out various operations. This tool is free to use and has some fairly complete tutorials that allow starting from the use of language to perform complex text analysis operations.
• SpaCy [52]: This toolkit features a variety of possible operations and claims to be designed for real-world jobs. It can be used in various languages as APIs and has some precompiled features. It also seeks to have a speed of operations at the current state-of-the-art level. In the same way as the previous ones, this is a free-to-use toolkit.
• Stanford POS Tagger [53]: This software is a Java implementation of the log-linear part-of-speech taggers described in [54]. Current downloads contain three trained tagger models for English, two for Chinese and Arabic, and one for French, German, and Spanish. The tagger can be retrained in any language, given POS-annotated training text for the language.

Figure 6 shows the available NLP techniques in each toolkit found. SpaCy NLP is the tool which includes more techniques, followed by NLTK and Stanford CoreNLP. Stanford POS Tagger is a specialized toolkit focused on POS Tagging and tokenization.

4.4. (RQ3) How Is the Performance Measurement of the NLP Techniques When Applied to Support the Documentation Process of an Agile Project?

With the analysis of the studies, we found different approaches in the use of diverse NLP techniques. Depending on the approach in which each technique is explored, the performance of the technique varies. Due to the mentioned above, a fair comparison is difficult since the techniques are not performing the same activity. Still, some methods are used to assess performance. Most of the studies report evaluation methods based on study cases or using prototypes to demonstrate the feasibility of their proposal. Some studies show at least a theoretical proof of concept to validate their proposals. Some other studies use precision, recall, and F-measure measures as indicators. The precision is about “What proportion of identifications was correct?” and it is calculated as it is shown in Equation (1). Recall refers to “What proportion of actual positives was identified correctly?” and it is calculated as it is shown in Equation (2). Finally, F-measure unifies the precision and the recall calculating as it is shown in Equation (3).

$$Precision=\frac{TruePositive}{TruePositive+FalsePositive}$$

(1)

$$Recall=\frac{TruePositive}{TruePositive+FalseNegative}$$

(2)

$$F-Measure=2x\frac{Precision\times Recall}{Precision+Recall}$$

(3)

where $TruePositive=$ the correct labeled instances $FalsePositive=$ incorrectly labeled instances, $FalseNegative=$ the missed-out instances by the system.

Mainly, performance evaluations on the selected papers were made by comparing the results obtained by humans with those obtained by a system. Usually, a group of volunteers is used as annotators who can be students or software developers, depending on the authors.

From the total 56 articles, only 12 use Precision/Recall/F1 as methods to validate the proposal’s performance.

5. Discussion

This work shows the use of NLP in the agile documentation process. The capacity of NLP for data processing, and the characteristic of requirements written in natural language to be understandable by all the project team stakeholders, make it possible to use NLP to analyze and process requirements artifacts. This process produces valuable information obtained from the artifacts to avoid human errors and making easier the analysis of the extracted information. Some related studies address the analysis of user stories using NLP [46], but they are not focused on improving the documentation process of an agile project. Other studies are focused on the general use of intelligent tools in agile methodologies [47] without emphasizing the possible use of NLP to bring to that tools the capability of extracting meaningful information from artifacts to improve performance. On the other hand, in [48], the importance of documentation in natural language in agile methodologies is discussed, and they concluded that using natural language in documentation is widely used by organizations. However, they did not consider or analyze how NLP tools could take advantage of it to extract relevant information from projects’ documentation.

Below, we present discussion corresponding to each research questions divided by subsections. Section 5.1 refers to RQ1, Section 5.2 refers to RQ2, and Section 5.3 refers to RQ3.

5.1. Approaches to Improve the Agile Documentation Process Using NLP Techniques

The selected articles are focused on improving the documentation process; most of them prefer to focus on processing requirements artifacts or requirements representations without distinguishing in which agile methodology (see Section 4.2). This situation allows us to adapt the tools to any methodology by not mixing something specific of some methodology in the tool. For example, user stories were the most common artifacts in the works, which can be applied in various agile methodologies. On the other hand, in some cases, better performance could be achieved by taking into account the context of an artifact within the internal process of a specific methodology. Other artifacts related to a particular methodology could also enhance the performance of the developed tool. For example, if the Scrum context is considered, we could consider the sprints and the backlog information [22] to enhance the quality of the data used to generate or transform artifacts. More studies may be needed to explore the impact of context with NLP. We found several ways in which NLP is used within the agile project documentation process. Some of them focus on processing user stories [46], while others choose to use other artifacts [48] such as use cases, use case scenarios, goal models, and class diagrams.

5.2. NLP Techniques and Toolkits Used to Improve the Documentation Process of Agile Development

During the analysis of the use of NLP techniques in the selected articles, we found that POS Tagging was the most commonly used technique. This finding indicates that tagging elements in a text is an important task when applying NLP to agile requirements. POS Tagging helps to identify each part of a text and allows applying other types of semantic analysis techniques, dependencies, and correlations, among others. NLP techniques, such as Tokenizer (the second most used technique) are commonly used as a preprocessing step; this indicates that using a preprocessing step might deliver better final results.

We also found that using NLP toolkits is entirely accepted within the studied area. It allows faster progress with research ideas to turn them into tools by making the development process easier and providing practical tools when working with NLP. These toolkits eliminate the need to recreate from scratch a process that effectively uses the desired NLP technique and provides a field-proven and accepted technique. Thus, researchers can move forward more confidently, knowing that what they use to develop a tool has been successfully tested multiple times. Although toolkits are widely used, in some cases, it might be better to develop the necessary NLP technique from scratch to combine it efficiently with other deep learning methods such as convolutional networks [55]. Some studies focus on comparing different toolkits applied to the same dataset of words [33]. These studies allow researchers to identify the appropriate toolkit depending on the task they want to address.

5.3. Performance Measurement of NLP Techniques Applied to Support the Documentation Process of Agile Development

Regarding the answer to question RQ3, we noticed that there is no standardized method to evaluate the performance of an NLP tool in agile applications. We found that each reviewed article presented an evaluation at their convenience. Either they are qualitative, based on the perception of those who used the tool, or quantitatively, using some method such as precision, recall, or F-measure indexes. The lack of a standardized method causes many of the results to be not comparable. These comparison difficulties can be a problem in determining the best-performing tool or evaluating a new tool with existing ones in the literature. Suppose some authors establish and require a standardized method for performance evaluation. In that case, the new approaches to NLP applications could be easily compared with the other existing approaches to show their characteristics and the best performance. In ref. [34] are described some evaluation methods recommended for NLP systems; the authors established that, although humans do the evaluation, it is essential to condense it into a standard measurement and not leave it only in data given by human evaluators. This lack of standardized methods for performance evaluation may indicate that the research area is still in progress and is on its way to obtaining a better maturity level in future years.

5.4. Threats to Validity

This study has some threats to validity that could affect the results or conclusions. We present them in the following four categories established in [56] and recommendations explained in [57].

5.4.1. Internal Validity

This category includes threats that may have affected the results and need to be properly considered.

• Research Method: One threat to validity is the restricted access to some databases that were excluded from the selection. To overcome this threat, we collected papers from databases used frequently in software engineering research [46]. Another threat could be improper research methods, which we overcome using the guidelines proposed by Petersen et al. [32]. The different search methods of each database were also a threat mitigated with the design of a base search string adapted to each database.
• Research Questions: The set of defined research questions for this study might not cover all the possible interest areas in the field, but we defined them in a way that covers the main fundamental points of interest.

5.4.2. Conclusion Validity

This category includes the threats that may affect drawing inaccurate conclusions from observations.

• Selected articles: The article selection might be improper because of the researcher’s bias. This threat was mitigated by doing a pair search with the help of another researcher. The results were similar not only on the first search (with the search string), but also in the reviews of the title, abstract and full text.

5.4.3. Construction Validity

This category includes threats to the relationship between theory and observation.

• Analysis: Validity of construction is the threat that experimenters can influence the results of a study based on what they expect from their experience in the field. This threat is not present in our work because we only report data founded on the selected papers without changing them.

5.4.4. External Validity

This category includes threats that affect the generalizability of the results.

• Generalization: As Wohlin et al. said in [57], the generalization of the results to the search field is very important, this could be a threat to our study depending on which studies were selected. To mitigate this threat, we defined all the search processes in a general form that could bring us a variety of studies; this is, using distinct databases with the same search string and analyzing each result by abstract, title, and full text.

6. Conclusions and Future Work

This work presented a Systematic Mapping Review to show the existing research on applying NLP techniques and toolkits in the documentation process of agile development methodologies. Many NLP techniques have been used to improve agile development processes; Tokenizer, POS Tagging, Parsing, Word, and Semantic Analysis are the most used in the founded articles. In addition to specific techniques, toolkits that facilitate using these techniques have also been found, and new ones are currently being published. These findings show that the field is under continuous study, which indicates that there are still points of interest and areas of opportunity to research.

Although we do not find related works on the same subject, some similar reviews address documentation and NLP. Next, we present two examples of related papers with similar subjects. In [46], they review the research of NLP with user stories; they report how NLP has addressed user stories to write them, analyze them and check their quality. On the other hand, in [47], the use of intelligent software in the documentation process of agile projects is addressed by obtaining the most frequent areas of the software development process like software management, software design, testing, and quality assurance, in which intelligent software was used. The main differences with these works are that our work is not limited to a single artifact like user stories, even if they are widely used, and also is not limited to a particular way to apply NLP. Moreover, our work focuses on using NLP rather than other intelligence techniques.

Below, we present the conclusions and future work of this research.

6.1. Conclusions

Regarding NLP technique applications to improve documentation process focusing on artifact conversion, something that we have not found and that could be interesting is a complete tool that can process an artifact as input to obtain a user story as output. Moreover, at the same time, to be able to provide the developer with a code template serving as a base for developing the code fragment related to that user story. The closest examples to this approach were works that convert user stories into use case scenarios and use cases into goal models.

There are two main ways when working with NLP techniques: developing it from scratch or using a toolkit. Using personalized NLP techniques is an important option when wanting to work in the area. It must be considered that when a technique is developed from scratch, some essential points can be customized, not only the parametrization. For example, a better method can be established to combine the NLP technique with other deep learning techniques, such as CNN or LSTM neural networks. The preprocessing, processing, and post-processing methods can form a complete tool that, when developed from scratch, can be self-sufficient and a complete tool for a specific purpose. The problem is the time required to sufficiently test the custom tool in the desired language. This problem is avoided in toolkits because they already have support for various languages, and if required, it is easier to change the desired language. The number of toolkits available and their characteristics allow us to start working with NLP easily and quickly. Each toolkit has a variety of languages that can be processed with it. This variety of toolkits allows various valuable systems to be developed in languages like Spanish, English, Portuguese, French, and many others.

The need for standardized measuring methods shows that the need to establish a standardized method for evaluating performance must be addressed in future work to improve the comparison of approaches; moreover, to make more visible to researchers the strengths and weakness of each approach to applying NLP to agile documentation. Making this visible allows researchers to proceed more confidently by making appropriate comparisons with existing work in the literature without the need to seek to convert others’ data into a standard comparable measure for all. Even though there are previously established methods in the area of NLP for performance evaluation, only 12 used said methods in the relevant papers. The other works focus on an evaluation made by people and presented in tables without performing the calculations of measures such as Precision, Recall, and F-Measure.

6.2. Future Work

Our results show a trend in using NLP in agile methodologies in the previous years. This trend suggests that the field will continue to grow and is worth contributing to NLP in agile methods. The papers have focused more on working in the category of requirements engineering improvement, which is a fundamental point of the agile development process. These findings and the need for more work focused on mixing several of these categories inspire searching for ways to combine at least two categories to create more complete tools that are helpful in agile development. In future work, we will combine requirements improvement, agile software process, and artifacts transformation to develop helpful tools for developers to create, organize, and update documentation artifacts during the agile project process.