Surfing the Modeling of pos Taggers in Low-Resource Scenarios

Abstract

The recent trend toward the application of deep structured techniques has revealed the limits of huge models in natural language processing. This has reawakened the interest in traditional machine learning algorithms, which have proved still to be competitive in certain contexts, particularly in low-resource settings. In parallel, model selection has become an essential task to boost performance at reasonable cost, even more so when we talk about processes involving domains where the training and/or computational resources are scarce. Against this backdrop, we evaluate the early estimation of learning curves as a practical mechanism for selecting the most appropriate model in scenarios characterized by the use of non-deep learners in resource-lean settings. On the basis of a formal approximation model previously evaluated under conditions of wide availability of training and validation resources, we study the reliability of such an approach in a different and much more demanding operational environment. Using as a case study the generation of pos taggers for Galician, a language belonging to the Western Ibero-Romance group, the experimental results are consistent with our expectations.

1. Introduction

The application of machine learning (ml) techniques has significantly changed the landscapes of natural language processing (nlp) over the last decade, allowing some of the gaps derived from the use of rule-based approaches to be filled in, mainly its lack of flexibility and high development cost. Thus, although the state of the art makes clear that hand-crafted tools are not only easier to interpret and manipulate but also often provide better results [1,2,3,4,5], the high level of dependency on expert knowledge makes their implementation and subsequent maintenance costly in human terms, in addition to hindering their applicability to different languages [2,5,6,7,8]. Combined with the surge in computational power, the possibility of accessing massive amounts of data and the decline in the cost of disk storage, this has decisively contributed to the growing popularity of ml algorithms as the basis for classification [9] and clustering [10] models in a variety of tasks. This includes entity detection [11], information retrieval [12], language identification [13], machine translation [14], question answering [15], semantic role labeling [16], sentiment analysis [17] and text classification [18], among others.

At this juncture, although recent proposals based on deep learning (dl) have outperformed the traditional ml methods on a variety of operating fronts, the approach has also shown its limitations. Specifically, there are two main reasons for the popularity and the arguable superiority of the dl solutions: end-to-end training and feature learning—while end-to-end training allows the model to learn all the steps between the initial input phase and the final output result, feature learning offers representability to effectively encode the information in the data. However, the latter translates into the inability to handle directly symbols [19]. This implies that data must first be converted to vector representations for input into the model and then do just the reverse with its output, which leads to a complex interpretability of models. Other well-known challenges are the lack of theoretical foundation [20], the difficulty in dealing with the long tail [21,22], the ineffectiveness of inference and decision making [23], and the requirement of large amounts of data and powerful computing resources that may not be available. This complex picture looks even worse in the nlp domain [24], particularly when it comes to dealing with low-resource scenarios—i.e., languages, domains or tasks lacking large corpora and/or manually crafted linguistic resources sufficient for building nlp applications— [25]. On the one hand, feature-based techniques lead to an imperfect use of the linguistic knowledge. On the other, the scarcity of training data is not only problematic per se, but also because of its impact on the rest of trials. So, generating high-quality vector representations remains a challenge [26] and the imbalance in the training samples that start the long tail and bias phenomena is more likely, together with the proneness to overfitting of dl models [27], which can result in poor predictive power, thereby compromising both inference and decision making.

This has led to a renewal of interest in reviewing the role of dl techniques vs. traditional ml ones in the development of nlp applications [3,24,28,29,30], particularly in low-resource scenarios [25]. Special attention has been given here to sequence labeling tasks [30]. These encompass a class of nlp problems that involve the assignment of a categorical label to each member of a sequence of observed values, and whose output facilitates downstream applications, such as parsing or semantic analysis, so errors at this stage can lower their performance [31]. Among the most important, we can highlight named entity recognition [7,32], multi-word expression identification [29], and morphological [26] and pos tagging [2,33,34,35,36]. It is precisely in this framework, the generation of pos taggers for low-resource scenarios by means of non-deep ml, that we propose the study of model selection based on the early estimation of learning curves. With that in mind, we first overview the state of the art and our contribution in Section 2. Next, Section 3 briefly reports on the theoretical basis supporting our research. In Section 4, we introduce the testing frame for the experiments described in Section 5 and later discussed in Section 6. Finally, Section 7 presents our conclusions and thoughts for future work.

2. Related Work and Contribution

Model selection based on the estimation of learning curves has been the subject of ongoing research over recent decades, inspired by the idea that the loss of predictive power and of training are correlated [37]. In the scope of nlp, these techniques have been applied to the most commonly researched areas, such as, for example, machine translation. Specifically, they have been used here for assessing the quality systems [38,39], optimizing parameter setting [40], estimating how much training data is required to achieve a certain degree of accuracy [41] or evaluating the impact of a concrete set of distortion factors on the performance [42]. Their popularity is growing especially in the field of active learning—those iterative ml strategies that interact with the environment in each cycle, selecting for annotation the instances which are harder to identify—(al)  [43], where we can refer to applications for information extraction [44,45], parsing [46,47] and text classification [48,49,50,51]. The same is true for pos tagging [52,53,54] and closely related tasks, such as named entity recognition [55,56,57] or word sense disambiguation [58,59,60], always with the aim of reducing the annotation effort. Since al prioritizes the data to be labeled in order to maximize the impact for training a supervised model, it performs better than other ml strategies with substantially fewer resources. This justifies the interest in it as an underlying learning guideline to deal with low-resource scenarios [61,62,63,64] and specifically in the area of pos tagging [54,65,66,67,68,69].

Focusing on the early estimation of learning curves in al, we can distinguish between functional [55,70] and probabilistic [71,72,73] proposals, depending on the nature of the halting condition used to determine the end of the training process from the information generated in each cycle. As a basic difference, functional strategies not only permit the calculation of relative and absolute error (respectively, convergence) thresholds —an error (respectively, convergence) threshold measures the difference between the real and the estimated learning curves at a finite (respectively, infinite) approximation time, while the absolute or relative character is applicable to any type of estimation, referring in the first case to the strict difference between the values compared, and to that existing between values calculated during the prediction process in the second one— [74], but they are also simpler and more robust than techniques based on probabilistic ones. In particular, by replacing single observations with distributions, we introduce elements of randomness, and thus uncertainty. That way, to establish how much data is necessary to reliably build such distributions is no easy matter, and the same applies to rare event handling. Being related to the definition of a sampling strategy on a sufficiently wide range of observations, this question should be preventable or better dealt with in a functional frame [75], even more so when the scarcity of training resources make it difficult to apply probabilistic criteria.

In such a context, we face the evaluation of pos tagging models by early estimation of learning curves when working in resource-scarce settings, to the best of our knowledge, a yet unexplored terrain. Leading on from this and looking for an operational solution, we focus on al scenarios, turning our attention to a functional view of the issue. With a view to exploring the practicality and potential of the approach, we address it in a setting used previously to demonstrate its effectiveness when the availability of resources for learning is not a problem. That way, we take up both the formal prediction concept and the testing frame introduced in [70], which also allow us to contrast the level of efficiency to be expected when the conditions for training and validation of the generated models are much more restrictive.

3. The Formal Framework

Below is a brief review of the theoretical basis underlying our work, taken from [70]. From now on, we denote the real numbers by $\mathbb{R}$ and the natural ones by $\mathbb{N}$, assuming that $0\notin \mathbb{N}$. The order in $\mathbb{N}$ is also extended to $:=\mathbb{N}\cup \{\infty ,\mathbf{\infty }\}$, in such a way that $\mathbf{\infty }>\infty >i>0,\forall i\in \mathbb{N}$. Assuming that a learning curve is a plot of model learning performance over experience, we focus on accuracy as a measure of that performance.

3.1. The Notational Support

We start with a sequence of observations calculated from cases incrementally taken from a training data base, and organized around the concept of learning scheme [70].

Definition 1

(Learning scheme). Let $\mathcal{D}$ be a training data base, $\mathcal{K}⊊\mathcal{D}$ a set of initial items from $\mathcal{D}$, and $\sigma :\mathbb{N}\to \mathbb{N}$ a function. We define a learning scheme for $\mathcal{D}$ with kernel $\mathcal{K}$ and step σ, as a triple ${\mathcal{D}}_{\sigma }^{\mathcal{K}}=[\mathcal{K},\sigma ,{\left\{{\mathcal{D}}_i\right\}}_{i\in \mathbb{N}}]$ such that ${\left\{{\mathcal{D}}_i\right\}}_{i\in \mathbb{N}}$ is a cover of $\mathcal{D}$ verifying

$${\mathcal{D}}_1:=\mathcal{K}and {\mathcal{D}}_i:={\mathcal{D}}_{i-1}\cup {\mathcal{I}}_i,{\mathcal{I}}_i\subset \mathcal{D}\setminus {\mathcal{D}}_{i-1},∥{\mathcal{I}}_i∥=\sigma \left(i\right),\forall i\ge 2$$

(1)

with $∥{\mathcal{I}}_i∥$ the cardinality of ${\mathcal{I}}_i$. We refer to ${\mathcal{D}}_i$ as the individual of level i for ${\mathcal{D}}_{\sigma }^{\mathcal{K}}$.

That relates a level i with the position $∥{\mathcal{D}}_i∥$ in the training database, determining the sequence of observations ${\{[x_i,{\mathcal{A}}_{\mathbf{\infty }}\left[\mathcal{D}\right]\left(x_i\right)],x_i:=∥{\mathcal{D}}_i∥\}}_{i\in \mathbb{N}}$, where ${\mathcal{A}}_{\mathbf{\infty }}\left[\mathcal{D}\right]\left(x_i\right)$ is the accuracy achieved on such an instance by the learner. Thus, a level determines an iteration in the adaptive sampling, whose learning curve is ${\mathcal{A}}_{\mathbf{\infty }}\left[\mathcal{D}\right]$, whilst $\mathcal{K}$ delimits a portion of $\mathcal{D}$ we believe to be enough to initiate consistent evaluations of the training. For its part, $\sigma$ identifies the sampling scheduling.

In order to obtain a reliable assessment, the weak predictor generated at each learning cycle is extrapolated according to an accuracy pattern [70], which allows to formally compile a set of properties, giving stability to the estimates and widely accepted as working hypotheses by the state of the art in model evaluation [72,76,77,78,79,80].

Definition 2

(Accuracy pattern). Let $C_{(0,\infty )}^{\infty }$ be the C-infinity functions in ${\mathbb{R}}^{+}$, we say that $\pi :{\mathbb{R}}^{{+}^n}\to C_{(0,\infty )}^{\infty }$ is an accuracy pattern iff $\pi (a_1,\cdots ,a_n)$ is positive definite, upper bounded, concave and strictly increasing.

As a running accuracy pattern, we select the power law family $\pi (a,b,c)\left(x\right):=-a\ast x^{-b}+c$. Its use is illustrated in the right-most diagram of Figure 1 to fit the learning curve represented on the left-hand side for the svmtool tagger [81] on the xiada corpus of Galician [82], with the values $a=204.570017$, $b=0.307277$ and $c=99.226727$ provided by the trust region method [83]. Returning to the review of our notational support, we now adapt these calculation elements to an iterative dynamics through the concept of learning trend [70].

Definition 3

(Learning trend). Let ${\mathcal{D}}_{\sigma }^{\mathcal{K}}$ be a learning scheme, π an accuracy pattern and $\ell \in \mathbb{N},\ell \ge 3$ a position in the training data base $\mathcal{D}$. We define the learning trend of level ℓ for ${\mathcal{D}}_{\sigma }^{\mathcal{K}}$ using π, as a curve ${\mathcal{A}}_{\ell }^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\in \pi$, fitting the observations ${\{[x_i,{\mathcal{A}}_{\mathbf{\infty }}\left[\mathcal{D}\right]\left(x_i\right)],x_i:=∥{\mathcal{D}}_i∥\}}_{i=1}^{\ell }$. A sequence of learning trends ${\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]:={\left\{{\mathcal{A}}_{\ell }^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\right\}}_{\ell \in \mathbb{N}}$ is called a learning trace. We refer to ${\left\{{\alpha }_{\ell }\right\}}_{\ell \in \mathbb{N}}$ as the asymptotic backbone of ${\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$, where $y={\alpha }_{\ell }:=\underset{x\to \infty }{lim}{\mathcal{A}}_{\ell }^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\left(x\right)$ is the asymptote of ${\mathcal{A}}_{\ell }^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$.

The minimum level ℓ for a learning trend ${\mathcal{A}}_{\ell }^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$ is 3, because we need at least three points to generate a curve. Its value ${\mathcal{A}}_{\ell }^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\left(x_i\right)$ is our prediction of the accuracy for a case $x_i$, using a model generated from the first ℓ cycles of the learner. Accordingly, the asymptotic term ${\alpha }_{\ell }$ is interpretable as the estimate for the highest accuracy attainable. Continuing with the tagger svmtool and the corpus xiada, Figure 2 shows a portion of the learning trace with kernel and uniform step function $5\times {10}^3$, including a zoom view.

3.2. Correctness

Assuming our working hypotheses, the correctness of the proposal—i.e., the existence and effective approximation of a learning curve ${\mathcal{A}}_{\mathbf{\infty }}\left[\mathcal{D}\right]$ from a subset of its observations compiled in a learning scheme ${\mathcal{D}}_{\sigma }^{\mathcal{K}}$—is demonstrated from the uniform convergence of the corresponding learning trace ${\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$ [70]. Specifically, the function

$${\mathcal{A}}_{\infty }^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]:={\underset{i\to \infty }{lim}}^u{\mathcal{A}}_i^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$$

exists and is positive definite, increasing, continuous and upper bounded by 100 in $(0,\infty )$. In order to estimate the quality of our approximation, a relative proximity criterion is introduced. Labeled layered convergence, it evaluates the contribution of each learning trend to the convergence process in a sequence which is proved to be decreasing and convergent to zero. These layers of convergence can then be interpreted as a reliable reference to fix error (respectively, convergence) thresholds.

3.3. Robustness

Robustness is studied from a set of testing hypotheses, which assume that learning curves are positive definite and upper bounded, albeit only quasi-strictly increasing and concave. An observation is then categorized according to its position with respect to the working level (wlevel), i.e., the cycle after which irregularities would not impact the correctness. Considering that the learner stabilizes as the training advances and that the monotonicity of the asymptotic backbone is at the basis of any halting condition, it is identified as the level providing the first slope fluctuation below a given ceiling in such a backbone and, once passed, the prediction level (plevel) marks the beginning of learning trends which could feasibly predict the learning curve, namely, not exceeding its maximum (100) [70]. Based on this, wlevel is calculated as the lowest level for which the normalized absolute value of the slope of the line joining two consecutive points on the asymptotic backbone is less than a verticality threshold $\nu$, which is corrected by applying a coefficient $1/\varsigma$ for avoiding having to deal with to both infinitely large slopes and extremely small decimal fractions.

Definition 4

(Working and prediction levels). Let ${\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$ be a learning trace with asymptotic backbone ${\left\{{\alpha }_i\right\}}_{i\in \mathbb{N}}$, $\nu \in (0,1)$, $\varsigma \in \mathbb{N}$ and $\lambda \in \mathbb{N}\cup \left\{0\right\}$. The working level (wlevel) for ${\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$ with verticality threshold ν, slowdown ς and look-ahead λ, is the smallest $\omega (\nu ,\varsigma ,\lambda )\in \mathbb{N}$ verifying

$$\frac{\sqrt[\varsigma ]{\nu }}{1-\nu }\ge \frac{|{\alpha }_{i+1}-{\alpha }_i|}{x_{i+1}-x_i},x_i:=∥{\mathcal{D}}_i∥,\forall i such that \omega (\nu ,\varsigma ,\lambda )\le i\le \omega (\nu ,\varsigma ,\lambda )+\lambda$$

(2)

while the smallest $\wp (\nu ,\varsigma ,\lambda )\ge \omega (\nu ,\varsigma ,\lambda )$ with ${\alpha }_{\wp (\nu ,\varsigma ,\lambda )}\le 100$ is the prediction level (plevel).

Following our example, Figure 3 shows the scale of the deviations in the asymptotic backbone before and after wlevel, for $\nu =2\times {10}^{-5}$, $\varsigma =1$ and $\lambda =5$. The plevel is also displayed, proving that these two levels might not be the same.

4. The Testing Frame

Given a training corpus $\mathcal{D}$, we want to study how far in advance and how well a learning curve ${\mathcal{A}}_{\mathbf{\infty }}\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$, built from a kernel $\mathcal{K}$ and using a step function $\sigma$, can be approximated in a low-resource scenario. To ensure the relevance of the results obtained, we standardize the conditions under which the experiments take place, following the same criteria previously considered in the study of resource-rich languages [70].

4.1. The Monitoring Structure

As an evaluation basis, we consider the run [70], a tuple $\mathcal{E}=[{\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right],\wp (\nu ,\varsigma ,\lambda ),\tau ]$ characterized by a learning trace ${\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$, a prediction level $\wp (\nu ,\varsigma ,\lambda )$ and a convergence threshold $\tau$. We then apply our study on a collection of runs $\mathcal{C}={\left\{{\mathcal{E}}_i\right\}}_{i\in I}$, defined for a set of different learners. In order to avoid misconceptions due to the lack of uniformity in the testing frame, a common corpus $\mathcal{D}$, kernel size, accuracy pattern $\pi$, step function $\sigma$, verticality threshold $\nu$, slowdown $\varsigma$, look-ahead $\lambda$ and convergence threshold $\tau$ are used.

In practice, we are interested in studying each run $\mathcal{E}=[{\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right],\wp (\nu ,\varsigma ,\lambda ),\tau ]$ from the level in which predictions are below $\tau$, and which we baptize convergence level (clevel). So, once the plevel is found during the computation of the trace ${\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$, we begin to check the layer of convergence. When it reaches the threshold $\tau$, the trend ${\mathcal{A}}_{\mathrm{C}\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$ becomes the model for the learning curve ${\mathcal{A}}_{\mathbf{\infty }}\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]$, and the process of approximation is stopped.

For the runs $\mathcal{C}={\left\{{\mathcal{E}}_i\right\}}_{i\in I}$, monitoring is applied to the learning trends ${\left\{{\mathcal{A}}_{{\mathrm{C}\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_i}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\right\}}_{i\in I}$ on a finite common control sequence of levels for the training data base, which are extracted from an interval of the prediction windows ${\left\{[{\mathrm{C}\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_i,\infty )\right\}}_{i\in I}$ [70]. In each control level, the accuracy (Ac) and the corresponding estimated accuracy (EAc) are computed for each run using six decimal digits, though only two are represented for reasons of space and visibility.

4.2. The Performance Metrics

Our aim is both to assess the reliability of our estimates and their robustness against variations in the working hypotheses. To do so, we employ two specific kind of metrics [70].

4.2.1. Measuring the Reliability

We here differentiate two complementary viewpoints: quantitative and qualitative. In the first case, it is simply a matter of studying the closeness of the estimates and the actual learning curves, while in the second, the objective is to determine the impact of those estimates on the decision making about the performance of some models relative to others.

The Quantitative Perspective

A simple way of measuring the reliability from this viewpoint is through the mean absolute percent error (mape) [84]. For every run $\mathcal{E}$ and level i of a control sequence $\mathcal{S}$, we first compute the percentage error (pe) as the difference between the EAc calculated from ${\mathcal{A}}_{{\mathrm{C}\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\left(i\right)$ and the Ac from ${\mathcal{A}}_{\mathbf{\infty }}\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\left(i\right)$. We can then express the MAPE as the arithmetic mean of the unsigned PE [70], as 

$$\mathrm{P}\mathrm{E}\left(\mathcal{E}\right)\left(i\right):=100\ast \frac{[{\mathcal{A}}_{{\mathrm{C}\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}}}^{\pi }-{\mathcal{A}}_{\mathbf{\infty }}]\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\left(i\right)}{{\mathcal{A}}_{\mathbf{\infty }}\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\left(i\right)},\mathcal{E}=[{\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right],\wp (\nu ,\varsigma ,\lambda ),\tau ],i\in \mathcal{S}$$

(3)

$$MAP\mathrm{E}\left(\mathcal{E}\right)\left(\mathcal{S}\right):=\frac{100}{∥\mathcal{S}∥}\ast \sum _{i\in \mathcal{S}}\left|\mathrm{P}\mathrm{E}\left(\mathcal{E}\right)\left(i\right)\right|$$

(4)

Intuitively, the error in the estimates made over a control sequence is, on average, proportional to the mape, which fulfills our requirements at this point.

The Qualitative Perspective

To that end, having fixed a collection of runs $\mathcal{H}$ working on a common corpus and a control sequence $\mathcal{S}$, the reliability of one such run depends on the percentage of cases in which its estimates do not alter the relative position of its learning curve with respect to the rest throughout $\mathcal{S}$. In this sense, our primary reference is the reliability estimation (RE) of two runs $\mathcal{E},\tilde{\mathcal{E}}\in \mathcal{H}$ on $i\in \mathcal{S}$, defined [70] as

$$\begin{array}{c}\mathrm{R}\mathrm{E}(\mathcal{E},\tilde{\mathcal{E}})\left(i\right):=\left\{\begin{array}{c}1\mathrm{if}[[{\mathcal{A}}_{\mathbf{\infty }}-{\tilde{\mathcal{A}}}_{\mathbf{\infty }}]\ast [{\mathcal{A}}_{{\mathrm{CLevel}}_{\mathcal{E}}}^{\pi }-{\tilde{\mathcal{A}}}_{{\mathrm{CLevel}}_{\tilde{\mathcal{E}}}}^{\pi }]]\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right]\left(i\right)\ge 0\hfill \\ 0 oth\mathrm{e}rwis\mathrm{e}\hfill \end{array}\right.\end{array}$$

(5)

with $\mathcal{E}=[{\mathcal{A}}^{\pi }\left[{\mathcal{D}}_{\sigma }^{\mathcal{K}}\right],\wp (\nu ,\varsigma ,\lambda ),\tau ]$, $\tilde{\mathcal{E}}=[{\tilde{\mathcal{A}}}^{\pi }\left[{\mathcal{D}}_{\tilde{\sigma }}^{\tilde{\mathcal{K}}}\right],\wp (\nu ,\varsigma ,\lambda ),\tilde{\tau }]$ and $\mathcal{E}\ne \tilde{\mathcal{E}}$. Having fixed a control level, this Boolean function verifies if the estimates for $\mathcal{E}$ and $\tilde{\mathcal{E}}$ preserve the relative positions of the corresponding observations, and it can be extended to the control sequence through the concept of the reliability estimation ratio [70].

Definition 5

(Reliability estimation ratio). Let $\mathcal{E}$ and $\tilde{\mathcal{E}}$ runs on a control sequence $\mathcal{S}$. We define the reliability estimation ratio (rer) of $\mathcal{E}$ and $\tilde{\mathcal{E}}$ for $\mathcal{S}$ as

$$\mathrm{R}\mathrm{E}\mathrm{R}(\mathcal{E},\tilde{\mathcal{E}})\left(\mathcal{S}\right):=100\ast \frac{{\sum }_{i\in \mathcal{S}}\mathrm{R}\mathrm{E}(\mathcal{E},\tilde{\mathcal{E}})\left(i\right)}{∥\mathcal{S}∥}$$

(6)

From this, we can calculate the percentage of runs in a set $\mathcal{H}$ with regard to which the estimates for a given one $\mathcal{E}$ are reliable on the whole of the control sequence $\mathcal{S}$ considered. We denote the resulting metric as decision-making reliability [70].

Definition 6

(Decision-making reliability). Let $\mathcal{H}={\left\{{\mathcal{E}}_k\right\}}_{k\in K}$ and $\mathcal{E}\notin \mathcal{H}$ a set of runs and a run, respectively, on a control sequence $\mathcal{S}$. We define the decision-making reliability (dmr) of $\mathcal{E}$ on $\mathcal{H}$ for $\mathcal{S}$ as

$$DM\mathrm{R}(\mathcal{E},\mathcal{H})\left(\mathcal{S}\right):=100\ast \frac{∥{\mathcal{E}}_k\in \mathcal{H},\mathrm{R}\mathrm{E}\mathrm{R}(\mathcal{E},{\mathcal{E}}_k)\left(\mathcal{S}\right)=100∥}{∥\mathcal{S}∥}$$

(7)

4.2.2. Measuring the Robustness

Since the stability of a run $\mathcal{E}$ correlates to the degree of monotony in its asymptotic backbone, we measure it as the percentage of monotonic elements in the latter through the interval $[{W\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}},{\mathrm{C}\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}}]$ where the approximation performs effectively. We baptize it as the robustness rate [70].

Definition 7

(Robustness rate). Let $\mathcal{E}$ be a run with asymptotic backbone ${\left\{{\alpha }_{\ell }\right\}}_{\ell \in \mathbb{N}}$, and ${\mathrm{C}\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}}$ and ${W\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}}$ its convergence and working levels, respectively. We define the robustness rate (rr) of $\mathcal{E}$ as

$$\mathrm{R}\mathrm{R}\left(\mathcal{E}\right):=100\ast \frac{∥\mu ∥}{∥\{{\alpha }_i,{W\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}}\le i\le {\mathrm{C}\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}}\}∥}$$

(8)

with μ the longest maximum monotonic subsequence of $\{{\alpha }_i,{W\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}}\le i\le {\mathrm{C}\mathrm{L}\mathrm{e}\mathrm{v}\mathrm{e}\mathrm{l}}_{\mathcal{E}}\}$.

The tolerance of a run to variations in the working hypotheses is therefore greater the higher its rr, thus providing a simple criterion for checking the degree of robustness on which we can count.

5. The Experiments

Within a model selection context and focused on the generation of pos taggers in low-resource scenarios, our goal is to provide evidence of the suitability of using evaluation mechanisms based on the early estimation of learning curves. It is a non-trivial challenge because, to provide that evidence, we need to study our estimates over a significant range of observations, which is in clear contradiction to the scarcity of training data.

5.1. The Linguistics Resources

In order to address the issue posed, we chose to work with a case study that meets four conditions. The first is that the language considered is really a resource-poor one, which should guarantee that it has been outside the tuning phase of the process of developing the taggers later used in the experiments, thereby precluding any potential biases associated with the learning architecture. Second, it should have a rich morphology, thus making the training process non-trivial and therefore relevant to the test performed. Thirdly, we should have at least a training corpus of sufficient size to study the reliability of the results obtained. Finally, that corpus should provide sufficiently low levels of convergence to allow the identification of the learning processes with the generation of viable models from a small set of training data.

We then take as a case study Galician, a member of the West Iberian group of Romance languages that also includes the better-known Portuguese. It is an inflectional language with a great variety of morphological processes, particularly non-concatenative ones, derived from its Latin origin. Some of its most distinctive characteristics are as follows [85]:

• A highly complex conjugation paradigm, with 10 simple tenses including the infinitive conjugate, all of which have 6 different persons. If we add the present imperative with 2 forms, non-conjugated infinitive, gerund and participle, then 65 inflected forms are associated with each verb.
• Irregularities in both verb stems and endings. Common verbs, such as facer (to do), have up to five stems: fac-er, fag-o, fa-s, fac-emos, fix-en. Approximately 30% of verbs are irregular.
• Verbal forms with enclitic pronouns at the end, which can produce changes in the stem due to the presence of accents: deu (gave), déullelo (he/she gave it to them). The  unstressed pronouns are usually suffixed and, moreover, they can be easily drawn together and often are contracted (lle + o = llo), as in the case of váitemello buscar (go and fetch it for him (do it for me)). It is also frequent to use what we call a solidarity pronoun, as che and vos, in order to let the listeners be participants in the action. That way, forms with up to four enclitic pronouns, such as perdéuchellevolo (he had lost it to him), are rather common.
• A highly complex gender inflection, including words with only one gender, such as home (man) and muller (woman), and words with the same form for both genders as azul (blue). Regarding words with separate forms for masculine and feminine, more than 30 variation groups are identified.
• A highly complex number inflexion, with words only being presented in singular form, such as luns (monday), and  others where only the plural form is correct, such as matemáticas (mathematics). More than a dozen variation groups are identified.

This choice limits the availability of curated corpora of sufficient size to a single candidate, xiada [82], whose latest version (2.8) includes over 747,000 entries gathered from three different sources: general and economic news articles, and short stories. With the aim of accommodating the elaborate linguistic structure previously described, the tag-set includes 460 tags, a short description of which can be found at http://corpus.cirp.gal/xiada/etiquetario/taboa (accessed on 13 July 2022).

5.2. The  pos Tagging Systems

As already argued, we focus on models built from al, selecting a broad range of proposals covering the most representative non-deep learning architectures—our reference here is the state of the art in pos tagging by the Association for Computational Linguistics (acl), available at the link https://aclweb.org/aclwiki/index.php?title=POS_Tagging_(State_of_the_art) (accessed on 13 July 2022)—, the same tested in [70] on resource-rich languages. This matching will allow us to establish, together with the subsequent identification of the parameters in the testing space, a valid reference based on the results obtained in that work:

• In the category of stochastic methods and representing the hidden Márkov models (hmms), we choose tnt [86]. We also include the treetagger [87], a proposal that uses decision trees to generate the hmm, and morfette [88], an averaged perceptron approach [89]. To illustrate the maximum entropy models (mems), we select mxpost [90] and opennlp maxent [91]. Finally, the stanford pos tagger [91] combines features of hmms and mems using a conditional Márkov model.
• Under the heading of other approaches, we consider fntbl [92], an update of the classic brill tagger [93], as an example of transformation-based learning. As a memory-based method, we take the memory-based tagger (mbt) [94], while svmtool [81] illustrates the behavior of support vector machines (svms).

In addition, this ensures an adequate coverage of the range of learners available in the computational domain under consideration.

5.3. The Testing Space

Following the method drawn by the choice of ml architectures discussed above, the design of the testing space is the same as the one considered in the state of the art [70] for the study of resource-rich languages, thus ensuring the reference value of the latter. Thus, in order to avoid dysfunctions resulting from sentence truncation during training, we retake the class of learning scheme then proposed, which permits us to reap the maximum from the training process. Given a corpus $\mathcal{D}$, a kernel $\mathcal{K}⊊\mathcal{D}$ and a step function $\sigma$, we build the set of individuals ${\left\{{\mathcal{D}}_i\right\}}_{i\in \mathbb{N}}$ as follows:

$$\begin{array}{c}{\mathcal{D}}_i:=⟦{\mathcal{C}}_i⟧,\mathrm{with} {\mathcal{C}}_1:=\mathcal{K}\mathrm{and} {\mathcal{C}}_i:={\mathcal{C}}_{i-1}\cup {\mathcal{I}}_i,{\mathcal{I}}_i\subset \mathcal{C}\setminus {\mathcal{C}}_{i-1},∥{\mathcal{I}}_i∥:=\sigma \left(i\right),\forall i\ge 2\hfill \end{array}$$

(9)

where $⟦{\mathcal{C}}_i⟧$ denotes the minimal set of sentences including ${\mathcal{C}}_i$.

Along the same lines and with respect to the setting of runs, the size of the kernels is $5\times {10}^3$ words, and the constant step function $5\times {10}^3$ locates the instances, which can be considered conservative values since smaller and larger ones are possible. Regarding the parameters used for estimating the prediction levels, the choice again goes to $\nu =4\times {10}^{-5}$, $\varsigma =1$ and $\lambda =5$. This also holds true for the selection of the regression technique used for approximating the partial learning curves and for $\pi$, which falls on the trust region method [83] and the power law family [95], respectively.

Taking into account that real corpora are finite, we study the prediction model within their boundaries, which implies limiting the scope in measuring the layers of convergence. We then adapt the sampling window and the control levels to the size of the corpus now considered. So, if $\lceil \lceil \ell \rceil \rceil$ denotes the position of the first sentence-ending beyond the ℓ-th word, the former comprises the interval $[\lceil \lceil 5\times {10}^3\rceil \rceil ,\lceil \lceil 7\times {10}^5\rceil \rceil ]$, whilst the latter are taken from control sequences in $[\lceil \lceil 3\times {10}^5\rceil \rceil ,\lceil \lceil 7\times {10}^5\rceil \rceil ]$. In order to confer additional stability on our measures, we apply a k-fold cross validation [96] to compute the samples, with $k=10$.

6. Discussion

As mentioned, the experiments are studied from two complementary points of view, quantitative and qualitative, according to the performance metrics previously introduced.

6.1. The Sets of Runs

To illustrate the predictability of the learning curves for the xiada corpus, we start with a collection of runs, $\mathcal{C}={\left\{{\mathcal{E}}_i\right\}}_{i\in I}$ generated from the data compiled in

Table 1. Monitoring of runs along the control sequences.

	plevel${}^{\mathbf{wp}}$	$\mathit{\tau }$	clevel${}^{\mathbf{wp}}$	Control-Level${}^{\mathbf{wp}}$										mape	dmr	rr
				$\lceil \lceil \mathbf{3}\times {\mathbf{10}}^{\mathbf{5}}\rceil \rceil$		$\lceil \lceil \mathbf{4}\times {\mathbf{10}}^{\mathbf{5}}\rceil \rceil$		$\lceil \lceil \mathbf{5}\times {\mathbf{10}}^{\mathbf{5}}\rceil \rceil$		$\lceil \lceil \mathbf{6}\times {\mathbf{10}}^{\mathbf{5}}\rceil \rceil$		$\lceil \lceil \mathbf{7}\times {\mathbf{10}}^{\mathbf{5}}\rceil \rceil$
				Ac	EAc	Ac	EAc	Ac	EAc	Ac	EAc	Ac	EAc
fntbl	105.003	2.40	150.017	94.16	93.87	94.57	94.30	94.96	94.61	95.16	94.84	95.34	95.03	0.32	85.71	90.00
maxent	110.047	2.50	135.019	92.90	92.78	93.30	93.19	93.58	93.48	93.85	93.70	94.08	93.88	0.15	100.00	100.00
mbt	85.012	2.20	145.016	92.97	92.84	93.42	93.22	93.76	93.50	94.01	93.72	94.30	93.89	0.28	100.00	92.31
morfette	75.011	2.60	105.003	94.61	94.54	94.98	94.89	95.21	95.14	95.41	95.33	95.55	95.49	0.09	100.00	85.71
mxpost	110.047	2.30	145.016	93.44	93.17	93.88	93.57	94.20	93.85	94.44	94.06	94.63	94.23	0.35	100.00	100.00
stanford	95.015	2.40	125.001	94.41	94.43	94.78	94.80	95.07	95.07	95.26	95.27	95.41	95.43	0.02	85.71	85.71
svmtool	250.012	2.20	250.012	95.00	95.05	95.36	95.44	95.60	95.71	95.78	95.93	95.92	96.10	0.12	100.00	86.67
tnt	85.012	2.00	130.003	94.47	94.38	94.79	94.70	95.05	94.93	95.23	95.10	95.35	95.23	0.12	71.43	100.00
treetagger	–	2.10	–	93.36	–	93.77	–	94.02	–	94.28	–	94.42	–	–	–	–

. The latter includes an entry for each one of the learners previously enumerated, together with its plevel and clevel, as well as the values for Ac and EAc along the control sequence, from which to calculate mapes, dmrs and rrs. In order to improve understanding, all the levels managed are indicated by their associated word positions in the corpus, which is denoted by using a superscript wp in their identification labels.

One detail that attracts our attention is that the run associated with treetagger does not reach the plevel within the limits of the training corpus. This behavior is certainly singular among all the taggers considered, which highlights the variety of factors that impact the evaluation of learners, and that, in this case, leads us to discard considering it in our study. In other words, in a real model selection process on the xiada corpus selected here, treetagger would not even be placed among the hypotheses that allow the application of the prediction technique considered.

6.2. The Quantitative Study

Our reference metric is now the mape, whose values are shown graphically in Figure 4 from the data compiled in Table 1. Taking into account that we are interested in numbers as small as possible, the scores range from 0.02 for stanford to 0.35 for mxpost in the interval $[\lceil \lceil 3\times {10}^5\rceil \rceil ,\lceil \lceil 7\times {10}^5\rceil \rceil ]$. Those results are illustrated in Figure 5, showing the learning curves and learning trends used for prediction on the runs with best and worst mape on the control sequence. As we have already done, the observations are generated considering the portion of the corpus taken from its beginning up to the word position indicated on the horizontal axis. Finally, 50% of mape values in this set of runs are in the interval $[0,0.12]$, a proportion that reaches 75% in $[0,0.28]$. Although these results are slightly worse than the ones reported in [70] for resource-rich languages, they are still very promising, which leads us to argue for the goodness of the proposal on the quantitative plane.

6.3. The Qualitative Study

Our reference metric is here the dmr, whose values are shown graphically in Figure 4 from the data compiled in Table 1. Taking into account that we are now interested in scores close to 100, these range from 71.43 to 100, with 85.71% of the values in the interval $[85.71,100]$. Moreover, the dmrs lower than 100 are the result of the intersection between the tnt learning curve with those of Stanford and fntbl. Under these conditions, the maximum value would only be possible if the error in the estimate of the intersection points was lower than the distance between its neighboring control levels, an unrealistic proposition, given how short that distance is (5000 words). In any case, the results are comparable to those reported in [70] for resource-rich languages, also meeting our expectations from a qualitative point of view.

6.4. The Study of Robustness

The reference metric is now rr, and we are interested in values as close as possible to 100, the maximum. The results are shown in Figure 4 from the data compiled in Table 1. While rr values range from 85.71 to 100, the latter is only reached in 37.50% of the runs. This percentage rises to 62.50% for rrs in the interval $[90,100]$. Overall, these results even exceed those reported in [70] for resource-rich languages, illustrating once again the good performance of the prediction model, this time against variations in its working hypotheses.

7. Conclusions and Future Work

Our proposal arises as a response to the challenge of evaluating pos tagging models in low-resource scenarios, for which non-deep learning approaches have often proven to be better suited. For this purpose, we reuse a formally correct proposal, based on the early estimation of learning curves. Technically described as the uniform convergence of a sequence of partial predictors which iteratively approximates the solution, the method acts as a proximity condition that halts the training process once a convergence/error threshold fixed by the user is reached, and has already demonstrated its validity when the availability of large enough learning datasets is not a problem. In order to ensure the reliability of the results obtained, we once again used the testing frame considered then, involving both quantitative and qualitative aspects, but also the survey of robustness against possible irregularities in the learning process.

Special attention was paid to the selection of a case study combining representativeness and access to validation resources, which is somewhat contradictory in the context under consideration. We then focus on Galician, a minority language of complex morphology, for which the collection of available training resources is reduced to a single corpus of sufficient size and quality to ensure both the validation phase and a rapid convergence process. This set of unique features allows us to simulate and evaluate short training sessions in a non-trivial learning environment, associating them with a language with important deficiencies in terms of computational resources. The results corroborate the expectations for the theoretical basis, placing the performance at a level similar to that observed in the state of the art for resource-rich languages for the same learners. This supports the effectiveness of the approach for the model selection considered and its suitability to low-resource scenarios, as initially argued.

To the best of our knowledge and belief, not only is this the first time that a proposal for estimating the performance based on the prediction of learning curves has demonstrated its feasibility in a framework of this nature, but it has done so without any type of prior specific adaptation. In other words, no operational limitations to the original conceptual design were observed. All this justifies the interest in highlighting the independence, both in terms of language and usage, of the technology deployed. A comprehensible way of doing it is extending our analysis, first to a broader set of languages in a variety of language families, and then to other fundamental and applicative nlp tasks, which establishes a clear line of future work.