Deep Neural Network Concept for a Blind Enhancement of Document-Images in the Presence of Multiple Distortions

Abstract

In this paper, we propose a new convolutional neural network (CNN) architecture for improving document-image quality through decreasing the impact of distortions (i.e., blur, shadows, contrast issues, and noise) contained therein. Indeed, for many document-image processing systems such as OCR (optical character recognition) and document-image classification, the real-world image distortions can significantly degrade the performance of such systems in a way such that they become merely unusable. Therefore, a robust document-image enhancement model is required to preprocess the involved document images. The preprocessor system developed in this paper places “deblurring” and “noise removal and contrast enhancement” in two separate and sequential submodules. In the architecture of those two submodules, three new parts are introduced: (a) the patch-based approach, (b) preprocessing layer involving Gabor and Blur filters, and (c) the approach using residual blocks. Using these last-listed innovations results in a very promising performance when compared to the related works. Indeed, it is demonstrated that even extremely strongly degraded document images that were not previously recognizable by an OCR system can now become well-recognized with a 91.51% character recognition accuracy after the image enhancement preprocessing through our new CNN model.

1. Introduction

In the modern lifestyle, digital cameras are used extensively. However, document images captured from digital cameras are of a much lower or poorer quality than the ones obtained from traditional scanners concerning an optimal performance of subsequent document processing apps such as character recognition systems (OCR) [1,2,3,4] or document-image-based document classification systems [5,6] or visual speech recognition [7]. Indeed, document images acquired from digital cameras (possibly integrated into smartphones, which are very pervasive) are generally corrupted by various distortions such as noise, blur, shadow, geometric deformations, etc.

Therefore, one crucially needs some form of robust pre-enhancement before such significantly distorted document images can be given to either OCR or document classification systems. This ensures an acceptable confidence level for meaningful use in such mentioned applications. Figure 1 roughly illustrates both the input (s) (i.e., an image or some images) and the output of the blind enhancement module. It obtains a color image as the input and returns an enhanced image as the output. Examples of image enhancement preprocessing forms for such digital-camera-acquired document images are the following ones, separately or combined or integrated: deblurring, denoising, contrast improvement, adjusting the brightness, etc.

Image enhancement is widely used in atmospheric sciences [8], astrophotography [9], medical image processing [10], satellite image analysis, texture synthesis, remote sensing [11], digital photography, surveillance, and various video processing applications [12,13].

Image enhancement can be explained by using the following formula (see Equation (1)):

$$S=T\left(D\right)$$

(1)

where $S$ is the target image, $D$ is the degraded input image, and $T$ is a nonlinear function, transforming the degraded input image into its related “original” non-distorted image. For finding the function $T$, we can define a function $Z$ in such a way that by minimizing it (i.e., by $Z$), the function $T$ will be converging to the required image enhancement function given in Equation (1). Thus, our target function $Z$ is defined by using the following formula (see Equation (2)):

$$\min Z=\Vert S_e-T{\left(D\right)\Vert }_2$$

(2)

where the $S_e$ is the expected or assumed original “clean “or non-distorted image.

On the other hand, it is well-known that the edges of or in character images are very important for a good performance on the OCR character recognition apps [14]. Therefore, we can add a so-called Sobel pre-filter to both sides of Equation (1). This results in the

$$Sobel\left(S\right)=Sobel\left(T\left(D\right)\right)$$

(3)

By adding Equation (3) to our target function (see Equation (2)), we obtain the following new formula (see Equation (4)):

$$\min Z=\Vert S_e-T{\left(D\right)\Vert }_2+\Vert Sobel\left(S_e\right)-Sobel{\left(T\left(D\right)\right)\Vert }_2$$

(4)

where $T$ is our target nonlinear function and $S_e$ is our expected original image. In Equation (4), we try to estimate the best-possible nonlinear function $T$ by minimizing our target function $Z$ described in Equation (4). This (i.e., Equation (4)) would be our loss function (while designing and training an appropriate deep convolutional neural network), which is used for estimating the nonlinear function $T$.

For estimating the function $T$, two different approaches exist, traditional methods and deep neural network concepts. Regarding the traditional methods, they essentially solve the problem via analytical and heuristic schemes [15,16]. In the second approach, such as the method proposed in this paper, convolutional neural network (CNN) models are used. CNN is a type of deep neural network which mainly uses, amongst others, convolution operations in its many layers. Deep neural networks (DNN) are essentially multi-layer artificial neural networks (ANN) [17] which are composed of four main parts or, better, functional bricks. These parts are convolution layers or filters, subsampling layers or filters, activation functions or layers, and “fully connected” neural network layers or blocks. Deep neural networks are, in essence, well-suited for performing a series of complex processing operations, e.g., estimating inverse filter(s), classification, denoising, contrast enhancement, etc. Previous studies have indeed proven that these types of networks can be used for denoising [18,19], contrast enhancement [20], and deblurring [21,22]. Thus, they have enough potential to reliably perform various image processing tasks.

The model we design in this paper involves CNN bricks, the concept of patches [23,24], the preprocessing Gabor and Blur filters [25,26,27], and finally the replacement of the repetitive hidden layers with residual blocks to increase convergence and the depth of network without adding complexity to it [28].

Section 2 briefly explains and discusses some related works regarding deblurring, denoising, contrast enhancement, and local light adjustment. Our novel CNN model is then described in Section 3. In Section 4, our model is then tested and compared or benchmarked with other selected relevant models while using or involving the same test dataset. In Section 5, concluding remarks are formulated, whereby the quintessence of the results obtained is briefly explained.

2. Related Works

The related works discussed in this section concern three different issues or distortions, which are all relevant and thus considered separately. These three different issues are: denoising, deblurring, and contrast or brightness enhancement. Although a given document image can be contaminated by a mixture of those distortions, they are addressed separately (i.e., in two functional modules) by our novel method.

2.1. Deblurring

Image blur is a significant distortion of document images. For example, images captured with moving or vibrating cameras may cause “motion blur”; the atmospheric turbulence can cause “Gaussian blur”, and “defocus blur” may be caused by the so-called lens aberrations [29].

The main problem of deblurring is the so-called deconvolution. Deblurring is an ill-posed problem. Thus, the final solution or clearing of the deconvolution should be optimized according to the specific type of image commonly used for deblurring. This type of optimization was introduced by Fergus et al. [30], and it was later extended by other authors who added and involved some ad hoc steps in providing an approximate solution [15,31,32,33]. For example, Levin et al. [34] added specific constraints to the problem to provide a correct solution to the so-called PSF (point spread function) problems. Therefore, using ad hoc steps and understanding well the precise nature of images at stake is crucial for finding appropriate solutions to the deconvolution problem.

Recently, CNN is used extensively in blind blur enhancement of images [35], and also most state-of-the-art methods are based on CNN [36]. The flexibility and power of CNN provides the possibility to improve the blur of an image blindly, especially in edges (Super Resolution). The CNN is also used in different forms such as hierarchical [37] and pyramid networks [38] to deblur the images. For example, one of the approaches involves Generative Adversarial Networks (GAN) in many related applications, such as super-resolution, blind denoising, and deblurring [39]. Xu et al. [40] have proposed such models to deblur text images.

Most of these related studies show some image quality improvement, but they are mainly capable of deblurring images with only a low magnitude of blur and mostly for the types Gaussian blur or mean blur. Therefore, a new model which can be used for other types of blurs such as motion blur is needed. Beyond that, in many cases, besides blur, other distortions may be present simultaneously.

2.2. Denoising

Various types of noises such as salt and pepper noise, Gaussian noise, Poisson noise, speckle noise, and many other fundamental noise types can contaminate document images and deteriorate the respective image quality [41].

Due to the ill-posed conditions of denoising, it is very important to know the original image’s statistical distribution. For example, some models have been introduced to emphasize the local area of pixel-like and total variation (TV) regularization [42]. Although those methods remove some noise from images, they also remove, unfortunately, some other good or important or needed details from images [43,44,45].

Recently, a patch-based approach is showing to be an up-and-coming solution to denoise images [46]. In most denoising schemes, the corrupted input image is decomposed by the denoiser into a set of patches, which will then be denoised separately, and later, those parts (i.e., denoised patches) are merged to create the final denoised image. One example of such a scheme is the so-called sparse coding method, which is trying to map each patch into a sparse coefficient matrix, in which most of the elements are zero or near to zero (sparse) [47,48].

A further scheme, a group method, works significantly better than the last described method, as most involved patches are Gaussian mixture models and have a multivariant correlation. Natural images are considered non-Gaussian, but they have one problem: they usually need more resources to be executed [46,49].

The idea of a patch-based solution comes from the fact they are inside (i.e., parts of a bigger) one image. We can find patches at different locations within the image, which can have similarities, and these similarities can be used by the denoiser to better reconstruct the original clean images. These similar patches are referred to in what is called non-local self-similarity (NSS). This method was first used in non-local mean denoising [50] and later, because of its precision and effectiveness, extended by different studies [46,49,51,52,53,54,55,56].

Denoising by using a convolutional neural network (CNN) was introduced by Jain et al. [57]. They argued that their algorithm or scheme achieved better results than traditional denoising methods such as total variation, wavelet approach, and other analytical techniques. Later, this CNN-based model was enhanced by introducing therein linear rectifier function(s) [58] and batch normalization [59].

Most of the above methods work well for white noise, but they hardly remove other kinds of noise such as salt noise, pepper noise, or speckle noise [53,60]. In addition, some of them introduce new distorting artifacts to the images (during the denoising process), which would need to be corrected or removed.

2.3. Contrast Enhancement and Local Light Adjustment

Most research defines contrast enhancement as adjusting the dynamic range of pixel intensity distribution for good contrast enhancement in images facing low contrast concerns. Contrast deterioration is mainly due to some external factors such as background light and environment.

Low-contrast images may reduce the visibility for digital users [61]. Most techniques for contrast enhancement are histogram-based or Retinex-based schemes [62]. In the histogram-based method, the main idea is that contrast-related pixel values globally or locally should have a Gaussian distribution. Therefore, these histogram-based techniques try to correct the pixel values’ distribution globally or locally according to a related window size definition [63]. Due to their simple functionality, they are easy to implement.

On the other hand, this histogram-based scheme in special situations hardly produces outstanding image quality, especially due to the complexity of image scenes and information loss in either low-contrast or high-contrast images. Further, they also eventually change the brightness of the processed image [64,65,66]. Therefore, several new related works have introduced new boundary conditions in the form of different regularization strategies.

For example, one can name the so-called Retinex-based method introduced by Land [67]. It is based on human color perception. The main concept of Retinex is based on composing illumination and reflections. The Single-Scale Retinex (SSR) is the implementation of center or surround Retinex. It provides reflectance as a result of the algorithm-like difference-of-Gaussian function. The Multi-Scale Retinex is similar to the previous one, but it contains a weighted sum of different SSR. Due to mapping color in an extra dimension, typically, after executing this algorithm, a color correction is required [62]. Both methods have also been implemented via convolutional neural networks, but the Retinex-based scheme has resulted in very low-contrast images [62].

Overall, a review of all related works shows a lack of good models that can significantly enhance document images containing certain distortion types (e.g., shadows, spotlights, etc.), and further, document images containing several distortion types simultaneously. Therefore, such a robust model that can face harshly distorted document images is very needed.

3. Our Novel Deep Neural Method for Blind Enhancement

The basic problem formulation is graphically presented in Figure 1, which essentially underscores the CNN deep neural model’s goal to be developed. However, for reaching the goal, it has been shown in the relevant literature that one single deep neural network cannot solve the complex problem at hand. Instead, a much more complex and modular deep neural architecture is needed. Indeed, the primary distorting artifacts that can be found in document images can be categorized into the following categories (see Figure 2):

• Blur problem(s), e.g., focus blur, Gaussian blur, motions blur, etc.;
• Noise problem(s), e.g., salt noise or pepper noise, depending on image sensor sensitivity;
• Contrast problem(s), e.g., shadows, spotlight, and contrast deficits.

For solving (removing) the mentioned distortions from a given document image, our overall model (see Figure 3) is designed with two sequential modules: (a) a document-image blur enhancement module, and (b) a document-image noise and contrast enhancement module.

The blur enhancement module is responsible for removing blur artifacts of the image. It has two types: motion blur and focus blur. The second module is responsible for eliminating or removing noise and contrast problems. The noise types can be salt and pepper noise and Gaussian noise. This module also improves contrast by fixing issues related to a spotlight, shadows, and overall image contrast enhancement.

Document-Image Deblurring (Module 1, see Figure 3) and Document-Image Joint Contrast and Noise Enhancement (Module 2, See Figure 3)

Figure 4 shows the structure of our overall CNN architecture for deblurring, as well as contrast and noise enhancement. As we can see from a comprehensive review of the relevant state of the art, the enhancement process is one of the most challenging tasks in image enhancement research. Thereby, it is always required to perform various additional processing steps before the CNN model can enhance images with an acceptable resulting or final image quality. Therefore, in this work, we introduce two different steps before providing the input document image to the core convolutional neural network model. The input image in our illustrative experiments has a size of 246 × 246 pixels.

First, we split the image into small patches, all with the size of 128 × 128 pixels. The patches have a little overlap for later enabling a better reconstruction of the original complete image. In sum, in our illustrative experimental example, we have 12 images (each one patch) used for the next step (i.e., four split image patches, each with its corresponding three RGB channels). In a second step, we involve a bank of preprocessing layers, in which we intentionally add blur effects, Gabor with different blur, and Gabor kernels.

This set of blurred images and Gabor images, along with the original patches, are taken as our basis for the CNN model’s input (see Section A of Figure 4). The main idea behind this complex preprocessing structure is the intention of pre-filtering non-appropriate data. Each filter that is applied removes some uninteresting/non-appropriate information/data. Consequently, this results in a smaller neural network structure, and the related model training requires less time as this technique shrinks the searching area [25,26,27,68,69]. Figure 5 illustrates the effective impact of different parameters of the Gabor filter on the original image. It is worth mentioning that the pre-filtering processors do not yet enhance the input image, they just place it in a new intermediate state which enables the consecutive processing by the CNN model to perform much better. This is confirmed by the good performance reached.

In the main model, the residual blocks are used instead of pure convolutional layers (Section B of Figure 4). The residual block is formed by the first convolution layer, which is aligning the number of layers for the last summing. The output of this layer is duplicated into main and shortcut tensors. The main tensors go forward into a set of convolutional layers and batch normalization and finally are added to the shortcut tensor. This helps to increase the convergence by encapsulating the complexity into the residual blocks.

Increasing the number of (parallel) preprocessing layers to more extensive convolution layers enables the deblurring images with a larger magnitude of blur. On the other hand, it requires more images (test samples) to train and fulfill the converging model requirement to decrease the goal function given in Equation (4).

All those (parallel preprocessing) layers are concatenated together to create our inputs for our deep neural network.

All convolutional layers used in Figure 4 use the Relu activation function. Indeed, the Relu activation function [70] knowingly displays an outstanding convergence rate when compared to other activation functions.

4. Model Training and Discussion of the Results Obtained

The model’s training strategy is the most challenging part, and we need to consider different types of problems such as blur, noise, and contrast in the complex training process. Both quality and quantity of the training samples help better adjust or determine the weights in the CNN models. Each submodule converges into the best parameter settings because of each submodule’s final respective defined task.

The three following modules are used for creating the basic datasets. These datasets are used for the extensive comprehensive training of our models (see Figure 4 and Figure 5):

(a)

Blur generator module: this module is responsible for adding blur artifacts to the standard dataset for both training and testing purposes. It contains three types of blurs. These three types are focus, motion, and blur based on the PSF library.

(b)

Noise generator module: this module is responsible for adding noise artifacts to the standard dataset. It contains three types of noise. These three types are Gaussian, salt and pepper, and speckle noise types.

(c)

Contrast generator module: this module is responsible for adding low contrast and brightness effects to the standard dataset. It contains two types of artifacts. The first one is the contrast effect, and the second one is the brightness effect.

Three different reference datasets are used to create the training, validation, and verification datasets. The first dataset library contains 60,000 images from Hradis et al. [71]. It consists of cropped images from different scientific papers without any artifacts. The size of cropped images is 246 × 246 pixels. The second dataset library is gathered by our own team. It contains 6000 documents images and scanned images of different official documents such as driving licenses and identity cards with large sizes of 4000 × 4000 pixels. Then, artificial artifacts using

Table 1. Different parameter settings for our three different datasets generating modules.

Artifact	Generator Parameters	Description
Gaussian Noise	${\mathrm{Image}}_{\mathrm{output}}={ \mathrm{Image}}_{\mathrm{Input}}+\mathrm{Norm}\left( \mathsf{\mu },\mathsf{\sigma }\right)$ $\mathrm{where} \mathsf{\sigma }\in \left[0,10\right], \mathsf{\mu } \in \left[-10, 10\right]$	Norm is a Gaussian random generator with mean value μ and standard deviation $\mathsf{\sigma }$.
Speckle Noise	${\mathrm{Image}}_{\mathrm{output}}={ \mathrm{Image}}_{\mathrm{Input}}*\left(1+\mathrm{Norm}\left(0,\mathsf{\sigma }\right)\right)$ $\mathrm{where} \mathsf{\sigma }\in \left[0,0.04\right]$	Norm is a Gaussian random generator with mean value $0$.
Salt and Pepper Noise	$\mathrm{Let} \mathrm{define} \mathrm{S} \in {\mathbb{Z}}^{\mathrm{m}, \mathrm{n}}\mathrm{and} \mathrm{P} \in {\mathbb{Z}}^{\mathrm{m}, \mathrm{n}} \mathrm{where} :$ $\mathrm{m}=\mathrm{input} \mathrm{image} \mathrm{width}$ $\mathrm{n}=\mathrm{input} \mathrm{image} \mathrm{height}$ ${\mathrm{s}}_{\mathrm{i},\mathrm{j}}\in \left\{0,1\right\},{ \mathrm{p}}_{\mathrm{i},\mathrm{j}}\in \left\{0,1\right\}$ ${\displaystyle \sum }{\mathrm{s}}_{\mathrm{i},\mathrm{j}}= \mathrm{o} * \mathrm{t}$ ${\displaystyle \sum }{\mathrm{p}}_{\mathrm{i},\mathrm{j}}= \mathrm{o}*\left(1-\mathrm{t}\right)$ $\mathrm{t} \in \left[0.475, 0.525\right]$ $\mathrm{f} \in \left[0,80\right]$ $\mathrm{o}=\mathrm{m}*\mathrm{n}* \frac{\mathrm{f}}{100}$ ${\mathrm{Image}}_{\mathrm{output}}={ \mathrm{Image}}_{\mathrm{Input}}*\mathrm{P}*\left(1-\mathrm{S}\right)+\mathrm{S}*255$	S and P are two matrices in which the elements can have either 0 or 1. The matrix S defines white pixels in the image, and the matrix P represents black pixels in the image. They participate together in generating salt and pepper noise: s is an element of matrix S; p is an element of matrix P; m is number of rows in matrix either S or P; n is the number of columns in matrix S or P; f and t are random numbers in the defined range which define noise level in percent and both salt and pepper ratios.
Contrast	${\mathrm{Image}}_{\mathrm{output}}={ \mathrm{Image}}_{\mathrm{Input}}.\mathrm{R}$ $\mathrm{where} \mathrm{R} \in \left[-0.5, 0.5\right]$	R is a real value from the given set of values
Brightness	${\mathrm{Image}}_{\mathrm{output}}={ \mathrm{Image}}_{\mathrm{Input}}+\mathrm{R}$ $\mathrm{where} \mathrm{R} \in \left[-128, 128\right]$	R is a real value from the given set of values
Focus Blur	${\mathrm{Image}}_{\mathrm{output}}={ \mathrm{Image}}_{\mathrm{Input}}{*\mathrm{Kernel}}_{\mathrm{Focus} \mathrm{Blur}}$ $\mathrm{where} \mathrm{KernelSize} \in \left\{1\times 1, 3\times 3, 5\times 5, 7\times 7, 9\times 9\right\}$	The Kernel matrix is defined for convolution with the original image
Motion Blur	${\mathrm{Image}}_{\mathrm{output}}={ \mathrm{Image}}_{\mathrm{Input}}{*\mathrm{Kernel}}_{\mathrm{Motion} \mathrm{Blur}}$ $\mathrm{where} \mathrm{KernelSize} \in \left\{1\times 1, 3\times 3, 5\times 5, 7\times 7, 9\times 9\right\}$	The motion blur kernel has one direction of 45 degrees

parameters are added to those images. This third dataset is a real-world one and is not generated by an artificial generator of distortions. Each document image is acquired under different distorting conditions such as noise, low contrast, spotlight, or shadows, which represent harsh real conditions. Based on the acquisition conditions of those documents, they have different various distorting artifacts, almost randomly. However, overall, we sort them into five quality categories or groups depending on how much the OCR readability (here: Tesseract OCR) is negatively affected. The five different quality categories are: very bad (under 60% OCR accuracy without enhancement), bad (60–70% OCR accuracy without enhancement), average (70–80% OCR accuracy without enhancement), good (80–90% OCR accuracy without enhancement), very good (90–100% OCR accuracy without enhancement). Document types contained therein are driving license, identity card, etc. For training purposes, the images are cropped in portions of 246 × 246 pixels.

These images of first and second datasets are mixed, and they are used as the source of images for our different artifact generators.

Consider creating a dataset for the training of the noise module. Here, one is adding noise to the original clean document-image samples from the original dataset to the first and second datasets. Hereby, one is adding different kinds of noise randomly, individually, or in a mixed form. The noise parameters used are shown in Table 1.

Consider creating a dataset for the training of the contrast and brightness module. Here, we add random contrast and brightness to the clean document images’ first and second dataset and truncate the contrast module’s pixel, exceeding the image threshold. The contrast and brightness parameters are shown in Table 1.

Consider creating a dataset for the training of the blur module. The blur module is the most advanced one. It contains various forms of blur: Gaussian, median, motion, and focus. With this, one adds different kinds of blur randomly, individually, or in a mixed form. The blur parameters are shown in Table 1.

The resized images described above are artificially distorted by adding blur for the first model’s module (see Figure 3) or artificially adding both contrast issues and noise for the second model’s module (see Figure 3). A set of 150,000 images have been used for training. In addition, for both verification and testing, datasets consisting of 40,000 images each were used.

In the first and second module, the ADAM [72] optimizer is used to train the model end-to-end. For a faster learning, the batch size of 128 images is selected. The learning rate of ADAM starts at 5 × 10⁻³, decreases by a decay rate of 0.95 after 100 epochs to improve the loss value, and finally stops at 1 × 10⁻⁵. The network is trained until it reaches 1000 epochs.

Equation (4) is used as loss function for training the model.

All developed CNN models (see Figure 3, Figure 4 and Figure 5) were implemented on a PC with Windows 10 Pro, Intel Core i7 9700K as CPU, double Nvidia GeForce GTX 1080 TI with 8 GB RAM as GPU, and 64 GB RAM. On this platform, the overall training of the developed model takes approximately 8 h.

The performance results obtained for each of the two modules of Figure 3 are compared with, i.e., benchmarked against, the selected best state-of-the-art models in their categories. That comparison was conducted based on two different metrics. The first metric is the peak signal-to-noise ratio (PSNR), and the second one is testing the amount of OCR-related errors after reading document images by Tesseract OCR [73] version 5.0.0. This last-mentioned source uses the peak signal-to-noise ratio, which is calculated through the following formula (see Equation (5)):

$$\mathrm{PSNR}=10 \log \left( \frac{{255}^2}{\mathrm{MSE}}\right)$$

(5)

where MSE is the mean square error obtained by comparing the original clean or target image (ground truth) and the output image obtained from the enhancement models. This equation expresses the following fact: the lower the MSE, the higher the PSNR. Therefore, by increasing the amount of image quality, the value of PSNR will increase. For a further “quantitative” assessment or testing, we process all output images by a well-known OCR system called Tesseract. Tesseract is a free, open-source OCR software. All the sample outputs (deblurred, denoised) images were tested for readability by the Tesseract OCR [73] version 5.0.0.

Regarding the evaluation metrics for testing with OCR, we used two metrics which are generally used for assessing text recognition performance: the character recognition accuracy (CRA) and the word recognition accuracy (WCA). The CRA is the percentage of the number of characters recognized correctly per total number of characters, and the WCA is the percentage of the number of words recognized correctly. The related studies show that the WCA metric is generally used to compare text recognition performance of various schemes [74,75], therefore we used this metric to show the performance of our model.

The effect of newly introduced components such as blur and Gabor filters are examined based on the reference model of Hradis et al. [71]. Each of the new filter arrays are being added/removed (e.g., Gabor, blur) from the base model to show the performance based on the absence/presence of the respective component (ablation study). The newly introduced component will stay if it can improve the model behavior.

The next two sub-sections (Section 4.1 and Section 4.2) comprehensively explain and discuss the performance results of the first module and those of the second module of Figure 3.

4.1. Performance Results of Module 1 (of Figure 3) for Document-Image Deblurring

In this sub-section, we compare the performance of Module 1 with a selection of well-known deblurring methods as presented in several recent papers from the relevant literature. Most of those methods from the literature are analytical, but only two of them are involving CNN.

In Figure 6, we present for illustration sample examples of deblurring results by Module 1. As we can see in these sample pictures, most of the output images have recognizable characters. However, their respective original images are not readable (by human eyes or by an OCR system).

In Figure 6, image (a) is the reference input blurred image, which has been, by the way, also used in several previous related studies. We can clearly see (through a qualitative visual inspection by human eyes) in Figure 6 that Module 1 produces a better-quality output image compared to those obtained by the other competing models from previous works (the two latest of them are CNN-based).

Table 2. Showing the result of ablation study for the blur enhancement.

Considered Deblurring Models	Character Recognition Accuracy (CRA) by the Tesseract OCR	Average PSNR
Blurred reference images	0	14.15
Our model without blur and Gabor filters	70.23	32.32
Our model without Gabor filters	81.32	32.76
Our model without blur filters	91.12	33.32
Our model, Module 1	94.55	33.85

shows the result of the ablation study. It is very clear that the base model improves while adding the new blur and Gabor filters.

For a further, this time “quantitative” testing, we now process all output images such as that in Figure 6 by a well-known OCR system called Tesseract (Figure 7 shows the output images obtained by Module 1; for the other models from the literature, the same input images are taken, and corresponding outputs are generated). Tesseract is a free, open-source OCR software. All the sample output (deblurred) images (shown in Figure 6) are tested for readability by the Tesseract OCR [73] version 5.0.0.

An overview of the evaluation results obtained in this comparative testing is shown in

Table 3. The character recognition accuracy (CRA) obtained for all deblurred images after testing by the Tesseract OCR software [73] version 5.0.0.

Considered Deblurring Models	Type of Deblurring Model	Character Recognition Accuracy (CRA) by the Tesseract OCR	Average PSNR
Blurred reference Images	-	0	14.15
Xu and Jia [76]	Analytical	0	20.12
L0 deblur [77]	Analytical	0	18.14
Cho and Lee [78]	Analytical	0	25.10
Zhong et al. [79]	Analytical	0	27.3
Chen et al. [16]	Analytical	0	28.4
Cho et al. [80]	Analytical	66.35	30.10
Pan et al. [16]	Analytical	92.48	33.50
Hradis et al. [78]	ConvNet	68.3	32.20
Neji et al. [39]	GAN	69.55	32.12
Our model, Module 1	ConvNet	94.55	33.85

. Indeed, a good readability by the OCR system expresses the better quality of the enhanced image (obtained from the deblurring). Finally, the output images are also compared in terms of PSNR to show the “objective” quality of output images. The total number of images used for testing each model for producing the results of Table 2 was 1500 images. Those images were produced based on 100 images from the image dataset of Hradis et al. [71]; the blur types and parameters used are defined in Table 1.

The results presented in Table 3 above clearly show that most deblurring models result in a very large number of errors and can therefore not be used for reliably deblurring document images for a later reading by an OCR system. Although still weaker compared to that of Module 1, the only previous method with acceptable performance is the one by Pan et al. [78]. The concept of that model is, however, an analytical model, and reaches a performance of 92.48% character recognition accuracy (see Table 2). Amongst all results, our model (Module 1) shows a very good and the best performance ever, which is 94.55% character recognition accuracy (CRA).

Compared to the other solely CNN-based model (by Hradis et al. [78]) from the list of competing approaches in the benchmarking (see Table 3), our novel CNN model (Module 1) performs significantly better (Module 1: character accuracy rate of 94.55%, versus the other CNN model by Neji et al. [39] having a CRA of 69.55%).

Concerning PSNR, we also can see that our model is the best one, as it keeps color more in white noise and cleans the image much better.

4.2. Performance Results with Regard to PSNR of Module 2 for Document-Image Noise Reduction and Contrast Improvement

In this sub-section, we provide and briefly discuss a set of illustrative performance results of Module 2. Figure 8 shows some selected inputs and outputs of the model of Module 2. As we explained previously, contrast and noise enhancement are blindly performed jointly in the second part of our global novel model, i.e., in Module 2 of Figure 3. Indeed, Figure 8 clearly shows that contrast and noise reduction are performed simultaneously.

For comprehensively measuring the performance of Module 2, we need an appropriate quality metric to better assess the efficiency of the “noise removal + contrast enhancement” module. For this purpose, we introduce and use a metric which was explained by Fu et al. [60]. This last-mentioned source uses the peak signal-to-noise ratio, which is calculated through the explained formula (see Equation (5))$.$

Concerning noise, we compare the performance of Module 2 with some related models published in recent papers from the literature. Hereby, we consider the so-called salt noise. The models from the literature were trained mainly for salt noise, but our model is also trained and tested for Gaussian noise.

Figure 9 shows the performance of our model compared to state-of-the-art models for denoising images. The original image contains peppers. One part of the peppers is zoomed in to show the sharp part of the peppers. That image is used as the original image, and a 50% noise level is added to that image. One can see how each model tries to reduce the noise level, whereby our model outperforms the other models when considering the sharpness of the respective enhanced image.

Table 4. Comparison of the PSNR values of the output images obtained from different denoising models involved in this benchmarking; each column corresponds to a different denoising model. The last column presents the PSNR performance of Module 2.

Test Image	Noise Levels in the Test Image	PSNR for DBA [81]	PSNR for NASNLM [82]	PSNR for PARIGI [83]	PSNR for NLSF [60]	PSNR for NLSF MLP [60]	PSNR for NLSF CNN [60]	PSNR for Our Model (Module 2)
Pepper	30 50 70	26.85 25.27 22.11	22.38 21.82 21.58	28.88 25.44 21.46	32.27 27.99 23.04	30.01 28.57 27.04	32.99 30.23 27.70	34.66 32.57 30.01
Lena	30 50 70	34.35 30.13 25.21	28.18 26.15 25.88	33.88 29.44 25.46	34.21 30.14 25.04	30.01 29.30 27.34	35.19 32.23 30.70	35.66 32.57 30.81
Average Over 11 Images from the Standard Test Images	30 50 70	31.79 28.27 24.38	27.07 26.38 26.98	30.86 27.47 23.87	32.28 29.28 25.09	29.77 28.09 26.36	33.35 31.34 29.15	33.76 32.57 29.80
Average Over 100 Images from Hradis et al. [78] Test Images	30 50 70	31.37 27.36 23.37	26.69 25.32 26.74	29.92 27.35 23.84	31.45 28.84 24.12	28.89 27.08 25.42	32.45 30.35 28.53	32.91 31.70 29.65

shows the PSNR performance of the different denoising models used by Fu et al. [60]. The images are the same as in the source Fu et al. [60] to make a comprehensive comparison possible. The number of test images were 11 images from the standard test images (“Pepper”, “Lena”, “Baboon”, “Bridge”, “Barbara”, “Fruit”, “Cameraman”, “House”, “Starfish”, “Monarch”, “Plane”) and 100 images from the Hradis et al. [78] test dataset. Some of the results of single images such as Lena and Pepper directly show the difference amongst the results obtained from the different involved models.

We can see (see Table 3) that our model provides the best PSNR results compared to the other models. For example, compared to the Fu et al. [60] model, our model shows a PSNR improvement ranging between 5% to 8%.

4.3. Performance Results with Respect to the OCR Performance of Module 2 for Document-Image Contrast and Brightness Enhancement

One of the sensitive problems while recognizing text through OCR is low contrast and brightness issues. An improvement of the text contrast makes the dark or highlighted parts more readable. For example, the input texts in Figure 10 are not recognizable by an OCR software, but after their contrast and brightness are adjusted, they can recognize text with 95% accuracy. Figure 10 shows that the second module is capable of significant contrast and brightness enhancement.

4.4. Performance Evaluation of Our Novel Global Model including “Module 1 + Module 2” (of Figure 3) for Blindly Enhancing Document Images Simultaneously Distorted by Blur + Noise + Contrast

After comparing comprehensively the performance of separated submodules in the previous sub-sections in the presence of different artifacts, we discuss in this section the overall performance of the global model (Module 1 + Module 2) compared to other competing related models in the presence of a mix of artifacts or distortions; this test context shall be considered as an extreme stress test of the performance of the best models available.

The dataset which is presented in this section consists of document images obtained by our team in harsh acquisition conditions (by cameras) (see Figure 11). This selection of 100 samples has been used for stress testing the different models. As one can see on this collection of images, different forms of distortions are present in the images, for example noise, contrast, and shadows; mostly, various distortions are simultaneously present in the images. (Note: since the images do partially contain some personal data, those related parts have been covered by a black rectangle for privacy reasons.)

The main reason for these harsh conditions is to provide strongly distorted images (i.e., contaminated by a mixture of distortions) in order to better stress test the robustness of the models involved in a comprehensive benchmarking (our novel model and a selection of the top best models from the relevant literature).

Figure 12 shows, for illustration, the performance of our global model in the presence of sample document images captured by a mobile phone camera. As one can see, the image quality is significantly enhanced so that the OCR system that could previously (on the non-enhanced input images) not detect even a single character can now (after enhancement) catch all the text information without any problem. Image (a) in Figure 12 is the document that has a mixture of distortions, namely motion blur, contrast, and noise problems. Image (b) in Figure 12 is an enhanced image through our novel global model. Image (c) in Figure 12 is the output of the OCR by using image (a), whereby none of the text in the image (a) is detected. The image (d) in Figure 12 is the output of the OCR by using the image (b) in Figure 12, whereby all the texts in this image are well caught/detected by the OCR without any significant error (Figure 12d).

Table 5. The character recognition accuracy (CAR) was obtained for all enhanced images using our own dataset after testing the Tesseract OCR software [73] version 5.0.0. For each test, 100 images are used for testing each model considered here from our own dataset of real-world document images.

Considered Image Enhancement Models	CRA in Different Image Quality (By the Tesseract OCR)
	Very Good	Good	Middle	Bad	Very Bad
Mix distorted (blur + noise + contrast) reference images	95.23	85.32	57.21	33.26	13.56
Pan et al. [16]	94.78	85.48	66.71	56.12	46.88
Neji et al. [39]	93.10	86.34	67.23	57.36	57.64
Pan et al. [16] + NLSF CNN [60]	95.40	87.83	82.53	80.73	77.53
Neji et al. [39] + NLSF CNN [60]	95.35	88.25	84.45	81.65	75.45
Our global model (Module 1 + Model 2, seeFigure 3)	97.12	98.52	95.15	92.13	91.51

shows the performance of our novel global model compared to the other related best models. For stress testing different models in this benchmarking, we used our dataset of real-world document images (obtained under harsh conditions) that mobile phone cameras have captured. One can see (see Table 4) that our model is about 13 percent better than the state-of-the-art model for deblurring combined with a state-of-the-art model for denoising. This test/benchmarking demonstrates and underscores the superiority of our novel model compared to a combination of the state-of-the-art best models. On the other hand, taken alone, the models developed solely for deblurring are not good enough for real-world hard cases; one can see that there are too many recognition errors in Figure 12c.

5. Conclusions

In this paper, we developed a new deep neural model to reliably and memory-efficiently deblur, denoise, and enhance the contrast of document images so that they become more readable for OCR software systems. This accuracy is achieved by sequentially combining two different enhancement modules, the first for deblurring and the second for jointly denoising and contrast enhancement. Each of these two named modules were appropriately trained with an appropriate specialized training set for the respective specific sub-tasks. This task separation shows an outstanding performance.

Our Module 1 model performs excellently in the blur part and outperforms other deblurring methods by generating the most recognizable text image. In the noise and contrast enhancement part, Module 2, the developed model shows a superb denoising effect compared to the previous salt and pepper denoising methods. It outperforms the best of those related models in the range of 5% to 8%. Our contrast is not comparable, as text contrast enhancement has a different meaning than other contrast enhancement types.

After enhancement, our global model’s output images are significantly more readable by an OCR system. This enhanced readability is reached even in cases of very strongly blur-distorted document images; we could see an accuracy improving from 13.56% to about 91.51%. A core limitation of the concept developed in this paper is that it is valid and has been validated so far only for document images and not for other types of images. Document images are different and specific compared to other more general image types. Thus, the specific image enhancement endeavor in the context of this paper has the sole purpose of making document images more readable by OCR systems. This is a very specific context and specification, which is not the same for all other types of images. For example, if a distorted marriage photo needs to be enhanced, the enhancement target does not necessarily match the ones of a document image. Indeed, an enhanced document image may still contain some form of dirt, especially in the background. However, the readability through an OCR of the text contained therein was significantly improved.

Therefore, possible future works of interest may relate to developing a novel global neural model which shall be optimized to various types of images as needed in various application contexts (examples of contexts: smartphone cameras, scene analysis or a self-driving car, video streaming through a smartphone, etc.).