Segmentation and Classification of Zn-Al-Mg-Sn SEM BSE Microstructure

Abstract

The microstructure of materials is shaped not only by their chemical composition, but also by the thermomechanical processes used during the processing of a specific piece. The correct interpretation of the microstructure gives a rich source of information. This consists of several related steps, such as segmentation. Successful segmentation enables the qualitative as well as quantitative analysis of the individual microstructure components. The current paper deals with the segmentation and classification of four basic microstructure components of the Zn-Al-Mg-Sn alloy system. This is attempted with the help of several image processing techniques, where thresholding is the main one used. The investigated samples are the cast and annealed Zn-Al-Mg-Sn alloy bulks. The input data for this analysis are the SEM BSE images. These were taken for all alloys with a varying Sn content, covering a significant area of each investigated sample at different zoom levels. A semiautomatic algorithm running under Matlab is introduced. It addresses several tasks, such as preprocessing, noise filtering and decision methods. For the individual procedures, the time requirements for their execution are also indicated.

1. Introduction

Industrially used materials are characterised by various properties. One of the forms of their investigation is modern electron microscopy, which offers coded information about the investigated substance in the form of a microstructure. It represents the individual material phases or components, including their mutual relationships. One of the most important aspects in the field of micrography is the presence of contrast differences and their physical origin. Overall, microscopic relationships have their direct counterparts at the normal macroscopic level, which the user can assess as strength, susceptibility to corrosion, etc. It is therefore natural that the user tries to create an algorithm that would, if possible, automatically examine and decode the necessary information content of microstructures. Each such piece of information is largely unique in electron microscopy, and therefore, the algorithms used in practice have their own special features and success rates. It is usually necessary to consider the appropriate ratio between the time needed to acquire the input data and the time needed to create a method for these data, as well as the time for the actual processing and evaluation.

Scanning Electron Microscopy (SEM)—The essential character of the image is determined by an electron gun, more specifically by the beam of electrons that this device emits. By setting the operational parameters of SEM, the operator controls the characteristics of the beam that interacts with the sample: energy, diameter, gun current, incident angle, etc. [1,2]. The electrons from the beam interact with the atoms of the sample through a number of physical processes described as scattering. From these interactions arise backscattered electrons (BSEs), secondary electrons (SEs) and X-rays, which carry information about the sample: topographical features, composition, crystal structure and local electric and magnetic fields [2,3,4,5]. Different types of materials and different imaging modalities place different demands on the sample preparation, even before the sample is introduced into the microscope [1,6,7,8,9]. There is a general recommendation that efforts should be devoted to the acquisition of the highest-quality data and less data processing than the other way around [10].

Contrast mechanism—By scanning the emissions from the interaction of the incident electron beam on the sample, a grayscale image is created, which is characterised by the information with a certain contrast [5]. The successful interpretation of the information seen is conditional on understanding the physical events that gave a rise to it. The overall contrast is made up of the following components with a varying degree:

• topographic contrast—dominantly the SE signal, which is more sensitive to the 3D topography of the surface of the scanned sample;
• composition contrast—dominantly the BSE signal, which is more dependent on the atomic number of the individual atoms in the sample;
• channelling contrast—electrons from the beam penetrate more easily along the crystal lattice than in a random orientation.

The phase represents a homogeneous body of matter physically different from its surroundings in the investigated microstructure. Each pure metal solidifies with a specific crystal lattice while some of them, with a polymorphic nature, can also create different crystal structures on specific conditions (pressure, temperature). When pure metals are combined in alloys, additional crystal structures are formed in certain compositions and stable temperature and pressure ranges [11].

The microstructure of metallic materials is formed by the distribution and topological arrangement of the grains, phases, their interfaces and other defects in three dimensions. It is a direct result of the chemical composition of the alloy and the production procedure applied. This is most often represented in the form of heat, mechanical or chemical treatment [12,13]. To distinguish the microstructure from the macrostructure of metals, a magnification of typically at least 75× is defined [14]. The qualitative analysis of the microstructure depends to a large extent on the knowledge and experience of the observer. When processing microstructures, there is a general effort to minimise or completely exclude the interactive intervention of the operator [15] for the following reasons [16,17,18,19]:

• time and effort requirements;
• subjective errors (bias) and related repeatability;
• need for domain knowledge in the form of technological expertise.

Overall, the methods of microstructure processing can be divided into the following:

• manual [15];
• semiautomatic (it is possible to use image processing (IP) techniques with at least partial success and then apply a manual refinement—this is the intermediate step to cope with the data-hungry demand of deep learning (DL) [20]);
• automatic.

It is impossible to develop a universal algorithm for the analysis of all microstructures [10]. The method of work will be influenced mainly by the sample preparation, observation technique (microscope, mode of operation), and nature of the material. The microscopic image is characterised by its quality, the assessment of which has an objective and subjective side [21]. In addition to the contrast differences, noise and the various distortions are included in the particularities of the microscopic image [22]. It is noise that is the fundamental problem in extracting meaningful information from SEM images [23,24]. More details about the relevant models and noise types in the typical imaging modalities are given by [25], while noise in SEM images is explicated in [26,27]. From a practical point of view, the level of noise in SEM images can be significantly influenced mainly by the scanning speed [26]. A necessary feature in noise removal is edge preserving [28], but sometimes a blurring or introduction of the artifacts near the edges cannot be avoided [29,30,31]. Bonnet [32] provides an overview of the IP applications in microscopy in the form of nine trends, selected from which are the following:

• 1st trend—signal processing: noise removal and texture analysis;
• 5th trend—collaborative microscopy: complementary imaging modalities;
• 8th trend—semi-automation: creating training data for DL;
• 9th trend—interdisciplinary cooperation: assessing the properties of image material.

Image segmentation is a process of partitioning an image into different regions that are homogeneous or ‘similar’ in a certain sense. The segmentation should be complete, the regions continuous and maximal and they should be characterised by certain properties (intensity, colour, texture) [33]. Despite the large number of segmentation techniques, it has not yet been possible to develop a universal method for a wide variety of images [34]. Bhanu et al. [33] and Siddiqui et al. [35] list four basic groups of segmentation methods: thresholding, edge-based, region-based and clustering-based. The segmentation results can sometimes be improved by combining several different methods to create a hybrid method [36]. After the successful creation of segments, it is necessary to classify them.

Related works—When researching the sources dealing with the microstructures, it had been found that the given sample material was rather poorly represented in the publications. Likewise, IP of the microstructure images of similar alloy systems had been performed only in a limited number of publications—see

Table 1. Survey of the published sources.

Author	Year	Material	Information
Gogola et al. [37]	2021	Zn-Al-Mg-Sn	SEM, EDX, XRD identification of the phases
Ercetin et al. [16]	2021	Mg-Al-Sn	SEM, Zoom 500 sample: sanding, polishing, etching IP + manual corrections noise filter: normalised box filter, Gaussian filter
Chalusiak et al. [17]	2021	solid oxide fuel cells	focused ion beam (FIB) SEM several filters tested and set via particle swarm optimisation (PSO)
Li et al. [40]	2020	shale	EDX machine learning (ML)
Truglas et al. [39]	2020	Zn-Al-Mg	SEM SE, FIB tomography transmission electron microscope (TEM) noise filter: anisotropic diffusion filter
LeTrong et al. [41]	2014	limestone	X-ray microtomography noise filter: alternate sequential filter, mosaic operator pros and cons of thresholding histogram deepening

. Gogola et al. [37] and Gabalcova et al. [38] worked with the same sample material. The phase composition was determined using a combination of X-ray diffraction (XRD) and SEM with energy-dispersive X-ray (EDX). Using EDX and XRD, the solid solutions, as well as intermetallic phases of these alloys, were identified. Analysis of the Zn-Al-Mg microstructures is a challenge [39] as the resulting structures are often complex and require (may require) assessment at different zooms. There is no single analysis technique that allows these structures to be evaluated at all necessary scales.

Main contributions—In the current research, the focus is being put on the annealed samples, as their investigation in [37,38] indicated a significant influence of phase morphology on the corrosion behaviour of alloys. With the methods proposed in this paper, it is possible to:

• Understand the characteristics of the visual data from the modality SEM BSE (rich visual content). The chosen alloy has not been so far investigated through this approach (Sn content, zoom);
• Remove the bottleneck in the technological workplace—see Figure 1. The resolution, noise level and data acquisition time (compared to EDX) clearly speak in favour of the presented approach. The modality XRD is relatively fast, but it does not provide any visual data, just the numerical values of the phase quantification (PQ);
• Quickly process the sample and acquire a lot of information from a large surface. From these visual data, it is possible to find out more details regarding manifestation of the metallurgical process, chemical composition, semantics of the scene, class imbalance, etc.;
• Save time and energy for the preparation of training data for deep learning compared to the manual editing from scratch;
• Process the previously taken image documentation even if the sample does not exist anymore.

In the paper, the data from the modalities EDX and XRD are used only in order to confirm the results of the presented approach. Neither the modalities themselves (EDX, XRD) nor their data are necessary in actual operation. In this way, the classes identified by the microstructure segmentation and classification approach presented in the paper are to be the real ones in the sample.

The principal differences among all three modalities are depicted in

Table 2. Characteristics of the modalities SEM BSE, EDX, XRD.

Modality	Data Format	Resolution	Information	Acquisition Time
SEM BSE	visual data (2D matrix)	high	composition contrast	low
EDX	EDX data (3D matrix)	low	chemical elements	high
XRD	nonvisual data	-	chemical phases	-

. XRD confirms the correctness of chemical phases but does not provide any semantic context of a visual scene. The context for chemical elements is confirmed through EDX at lower resolution.

2. Materials and Methods

The processed data were the result of the observations from the previous research [37]. The image documentation was available for each alloy from about 50 locations at various zoom levels.

Table 3. Summary description of the task.

Substance	Metal alloy Zn-Al-Mg-Sn: Zn bal. wt.% (weight), 1.6 wt.% Al, 1.6 wt.% Mg, 0–3 wt.% Sn
Sn content	0, 1, 2, 3 wt.%
Task	Segmentation and classification: level thresholding + morphology, histogram thresholding
Number of classes	3 or 4 (depending on Sn content)
Microscope	JEOL JSM 7600F
Electron source	Schottky field emission gun
Detector	BSE—SM-74280RBEI in composition mode
Image exposure time	6600 ms
Working distance	15 mm
Sample treatment	Grinding 280—4000 grid emery paper Polishing 3—0.25 µm ethanol-based diamond paste Cooling and lubrication with ethanol
Software	Matlab 9.11.0.1769968 (R2021b)
OS + Hardware	Win10 Pro 21H2 64 bit, i7–7820HQ, SSD, 64GB RAM
Image information	960 × 1280, 8 bit grayscale
Noise removal	NonLocal Mean (NLM) filter
Zoom	1000, 2000, 4000

summarises the most important information regarding the task; more details can be found in the individual parts of the paper.

Data acquisition—The samples were cast into a round Cu crucible with a diameter of 30 mm and depth of 20 mm. These samples were consequently annealed at 310 °C for 1 h. During metallographic preparation, the samples were cut along the cylinder axis and the resulting sections were embedded into the Buehler KonductoMet conductive resin. Grinding was performed on a set of emery paper up to a granularity of 4000. Subsequently, polishing was performed on a set of alcohol-based diamond pastes down to a grain size of 0.25 µm. Ethanol was used for cooling and lubrication during all steps of the metallographic preparation to minimise the chance of corrosion. The samples were investigated without etching to avoid the formation of any additional surface roughness. This would have adversely affected the chance of a successful segmentation. To limit electrical charge build-up during the investigation, a conductive carbon tape was used to connect the samples to ground (metallic part of the SEM sample stage).

The microstructure documentation was performed by the JEOL JSM 7600F scanning electron microscope (SEM, Jeol Ltd., Tokyo, Japan). The Schottky field emission electron source was operated at the accelerating voltage of 20 kV and 90 µA of the emission current. The samples were placed at the working distance of 15 mm. The backscattered electron detector was utilised for the microstructure documentation.

The microscope settings were optimised for efficient image documentation at rather low zooms of up to 4000. The wide objective lens aperture (110 µm), the probe current of 2 nA and the exposure time of only 6600 ms per image were set. A JEOL SM-74280RBEI BSE detector consisting of two semicircle sensors was utilised. Setting them into the composition mode enabled the creation of images based on the atomic number contrast with the sufficiently low noise. The relatively short exposure time enabled the regular documentation of an approx. 80 mm long section for each alloy.

The chemical composition analysis was measured via an EDX spectrometer (Oxford Instruments plc, Abingdon, United Kingdom) with an Oxford Instruments X-Max silicon drift detector.

During image processing, the following microstructure components needed to be distinguished: the Zn-based solid solution η(Zn), the Mg₂Zn₁₁ intermetallic phase and the particles formed by the η(Zn) + α(Al) eutectoid mixture; finally, the addition of Sn enabled the formation of the Mg₂Sn intermetallic phase particles (see

Table 4. Investigated phases and microstructural components.

	Component	Description	Label for IP
1	η(Zn)	solid solution	BG background
2	η(Zn) + α(Al)	eutectoid (mixture of the solid solutions)	BW black and white
3	Mg2Zn11	intermetallic phase	L light
4	Mg2Sn	intermetallic phase	D dark

and Figure 3).

The individual microstructural components were characterised through a local EDX analysis. Figure 2a shows a BSE image of an area with all key microstructure components present. The base matrix is formed by η(Zn) represented by the Zn map in Figure 2b. The Al map (Figure 2c) clearly indicates the presence of the η(Zn) + α(Al) eutectoid mixture particles, while the areas containing both Zn and Mg (Figure 2d)) indicate the presence of the Mg₂Zn₁₁ phase particles. Finally, the presence of Mg and Sn (Figure 2e) indicate the location of the Mg₂Sn phase particles. In this paper, two designations with the same meaning are used interchangeably based on the context of the current section: particle (materials science point of view) and shape (IP point of view).

Figure 3 shows a section from an original unfiltered microscopic image (3% Sn, Zoom4000) with the dimensions of 300 × 500 pixels—throughout the paper, the notation [rows × columns] is used as in Matlab. The BG background, the BW eutectoid and the L and D shapes are observed. The image shows a typical white rim at the interface of the BG background and D shapes. A similar rim can also be observed on the interfaces of the BG and L shapes as well as the BG background and BW eutectoid, but only to a lesser extent. This is characterised by the visible elevations in the line segment figures (see Figure 6—the edges of the span approx. 250 to 480). As another detail, BG contains the precipitates of α(Al) solid solution visible as the small dark spots. These are observed in the η(Zn) solid solution BG, as well as on the BG/L and BG/D interfaces. These particles were ignored due to their small size (approx. 50 to 100 nm) and negligible influence on the total phase composition. In the centre of the image, a BW shape is present at the bottom, which gradually increases in intensity values upwards. The given particle is therefore not cut perpendicularly but at an angle of less than 90 degrees. It gradually begins to shine through from the background. The D shapes can also appear with a high-intensity interior—see Figure 14 for more details.

Microscopic image and accompanying information—The acquired microscopic images had dimensions of 1024 × 1280 pixels. They consisted of two parts: the image information itself in the upper part 1:960 × 1280 (grayscale in 8-bit depth) and a black information bar 961:1024 × 1280. The following information was extracted using optical character recognition: zoom, date and time of the image taking, and scale bar. The third source of information was the name of the image where the Sn content had been entered by the operator.

Data characteristics—In principle, a SEM image is characterised by resolution; contrast; edge sharpness; signal-to-noise ratio (SNR); structure sizes, which depend on the zoom used; and properties of the sample. These characteristics may be different on each SEM image, although the same operating parameters of the microscope or sample preparation conditions could be applied [42].

The nature of the data themselves is evident from Figure 4, which represents the section 820:1280 from a row (see the whole row in Figure 5) at 0% Sn and Zoom4000. The original data provided from the microscope are shown in the blue waveform; the filtered data using the NLM filter are in red. It has to be noted that most of the ‘normal’ noise is caused by the imaging technique used; however, the more pronounced oscillations towards the low intensities (the points A, B and C) are caused by the Al-based precipitates. The precipitate A is located on the boundary of the light shape L, while B and C are in the background BG. This fact is also reflected in the SEM images. The precipitates are located either inside the background or on the boundary between the shapes and background. These will, however, be mostly ignored, as mentioned above, due to their low size and overall volume. The precipitates within BG are considered as part of BG. The NLM filter works in a way that the sharp transitions in the BW shapes are kept (see Figure 5).

From Figure 4, the following can be deduced for calculating the SNR:

$$SNR=\frac{\Delta S}{\Delta N}\cong \frac{0.9-0.8}{0.05}=\frac{0.1}{0.05}=2,$$

(1)

Ul-Hamid [1] recommends the SNR value of at least 5 for ample contrast.

The typical data features for the individual Sn percentages as observed on the Zoom4000 images are summarised in

Table 5. Typical data features, Zoom4000.

Sn [%]	Description
0	noise 0.05~0.08, contrast BG/L 0.1 [25.5 pixels] background rel. horizontal, multiple backgrounds unobserved white rim up to 0.08 amplitude in BG on the borders L shapes: rel. narrow and long, horizontal bottom + drift some background precipitates remained after noise removal BW shapes: texture with regular alternation of B and W
1	noise 0.05~0.08, contrast BG/L 0.05 [13 pixels] multiple backgrounds possible (drift from 3 to 5 pixels across the entire image) white rim up to 0.15 amplitude in BG on the borders with: BW, D, and another BG L shapes: wider and more oval D shapes: area smaller than L shapes, intensity value varies possible presence of D shapes with high-intensity interior, which are much larger in area essentially no background precipitates after noise removal BW shapes: texture with regular alternation of B and W; pure black; irregular texture
2	noise 0.05~0.08, contrast BG/L 0.08 [20 pixels] rarely multiple backgrounds (drift from 5 to 15 pixels across the entire image) white rim up to 0.2 amplitude in BG on the borders with: BW and D D shapes: area smaller than L shapes, intensity value varies rare presence of D shapes with high-intensity interior, which are much larger in area essentially no background precipitates after noise removal BW shapes: texture more regular than at 1% Sn but less than at 0% Sn
3	noise 0.05~0.08, contrast BG/L 0.05 [13 pixels] background rel. horizontal, multiple backgrounds unobserved white rim up to 0.2 amplitude in BG on the borders with: BW and D D shapes: area larger than at 1% Sn and 2% Sn, intensity value varies essentially no background precipitates after noise removal BW shapes: texture rel. regular like at 0% Sn; rarely pure black

. The variability of the data was considerable, but the presented characteristics applied in general. The images of the 0% Sn and 3% Sn alloys were the most uniform and similar in character as well as content and distribution of the individual shapes. The visual contents of the 1% Sn and 2% Sn alloys represented the biggest problem of segmentation and classification. There was a huge variation between the particle shapes as well as sizes within the individual classes (L, D, BW). Even the BG shapes were showing ‘multiple backgrounds’. This effect was caused by partial influence of the channelling contrast on the final image. Different crystallographic orientation of the individual BG shapes caused them to appear on the images with varying intensity levels.

Pre-processing—After having extracted the image information itself, 1:960 × 1280 in 8-bit grayscale depth of the uint8 type [0~255], it was converted to the single-precision floating point type 0~1. The transfer is part of the block 1 initial steps in Figure 7. In this paper, all independent variables appear on line segment figures in a single format with a range of 0~1, similarly to the noise or contrast values in the text or tables (see Table 5). Additionally, it is possible to indicate the corresponding value converted to the uint8 type in square brackets and expressed in pixels. The reason for calculations with the single-precision floating point type is to preserve details. After performing the necessary calculations, it is possible to convert them back to the uint8 type and save the image in the tiff format.

Noise and technical filter—The NLM filter operation is depicted in Figure 5. The original data from the microscope are blue; the filtration result is illustrated in red. The entire row for 0% Sn and Zoom4000 is shown. The presented segments are BG background, L and BW shapes (the columns approx. from 250 to 650). The filtering caused only slight intensity changes in case of the BW shapes. On the borders of the background, slight elevations towards the higher intensities are noticeable.

It can be seen that the individual segments (especially the background) are not completely horizontal, but are rather characterised by a bent bottom towards the lower intensities. These elevations and bent bottoms give an incentive to apply the technical filter.

The operation of the technical filter levelling is depicted in Figure 6. A complete column of an image (0% Sn, Zoom4000) is shown. The data after the basic noise NLM filtering are blue. When examining it, using the absolute value of the gradient (only the fact of the change is of interest and not its nature), its local extremes—maxima—are found (magenta line). Obviously, the small gradients are ignored and only the extremes that have passed the selection level are considered. The sections between such gradients are subsequently levelled. The arithmetic mean of all blue data from the given section in the respective direction, the row or column, is calculated. The result is shown with the dashed magenta line. Since both directions (the rows and columns) are perpendicular to each other in the image plane and the individual shapes are generally characterised by significantly different properties within these two directions, it is necessary to consider both of them. Therefore, the technical filter levelling is applied twice (see Figure 7)—first time for all rows (first image) and second time for all columns (second image). After averaging (the arithmetic mean of both images) and subsequent filtering, the result of technical filtering is available in the form of the red curve. It is clear from Figure 6 that such a result is between the original denoised waveform (solid blue) and the levelling in one direction (dashed magenta).

The purpose of the entire operation is obvious when assessing the distance of the individual blue bent segments to the imaginary green threshold compared to the more ‘horizontal’ courses of the same segments in red—the section, e.g., between rows 1 to about 500. In more demanding images (see Figure 11), the distance is quite small, or even disappears.

Table 6. Running times for the preprocessing, noise filter and technical filter.

	Figure 7	Running Times	Information
preprocessing	block 1	7 [ms]	im2gray, im2single
noise filter (Zoom4000)	block 2	60 [s]	3× NLM filter
technical filter	blocks 7,8,9,10	0.4 [s]	1× repetition
	block 11	20 [s]

shows the total typical running times of the individual steps: preprocessing, noise filter at Zoom4000 (influenced by the NLM filter specific value of SearchWindowSize (SWS)) and technical filter for an image 1:960 × 1280. As the value of SWS configuration parameter decreases—see blocks 2 and 11 in Figure 7—the running times of the given filter also decrease.

Decision methods—Figure 7 shows the overall overview scheme of segmentation and classification. The green blocks 1, 2, 3, 4, 5, 6 represent the procedure without applying the technical filter; the red blocks 7, 8, 9, 10, 11 represent just the technical filter. The black arrows between 1, 2, 3 are a common route for both procedures, and then either blue or red continue. Depending on the percentage of Sn content, block 6 is present or not. The purple dots indicate the places in the algorithm where the configuration values are present. These had been preset based on the expertise during the algorithm development. For most of the images, they worked sufficiently. In occasional cases, they could be manually tuned. The choice between the procedure A or B and how to tune the configuration values depends on the user’s experience.

Description of the individual blocks:

• Initial steps—the selection of the pure image information, its conversion to the single-precision floating point type and extraction of the zoom and %Sn information;
• NLM filter—the applying of several consecutive NLM filters to remove noise. Function: imnlmfilt. The incorrectly selected configuration values result in poor noise removal or, on the contrary, loss of important edges of shapes. It is therefore necessary to pay sufficient attention to this step. Configuration values: DegreeOfSmoothing (DOS), SWS, ComparisonWindowSize (CWS);
• BW—the classification of the first BW segment. The classification of the BW shapes can be divided first into level thresholding, when according to a priori statistically obtained value, which is sufficiently low, the dark BW components are identified. These are subsequently expanded using the morphological operations imfill and imclose with a suitably chosen structuring element, completed and closed with the white BW components. Configuration values: intensity level threshold to black, type and size of the structuring element;
• BG/Shapes—the separation and classification of the BG background from L + D shapes. Method: histogram thresholding. Configuration values: intensity histogram threshold, moving average (MA) filter parameters and number of sequential filtering;
• Corrections—the optional step where some pixels can be swapped between the BG segments and L + D Shapes (such as precipitates). Functions: bwconncomp to obtain the connected components with the default connectivity 8, regionprops(‘Area’) for obtaining the number of pixels in connected shapes. Configuration values: number of pixels in connected shapes;
• L/D—this step occurs only in case of nonzero Sn content. Method: histogram thresholding. The data, left over from the previous segmentations, are divided into the L and D shapes. Configuration values: intensity histogram threshold, MA filter parameters and number of sequential filtering.

Technical filter:

7.

BW—all values of the identified BW segment can be intensively adjusted to another level (e.g., 0). This will eliminate the problems with large and sudden intensity changes within the BW segment (see e.g., Figure 5 indices between 600 and 720). These would be difficult to process in the steps based on gradients—blocks 8 and 9. Configuration values: intensity level threshold to black, type and size of the structuring element, setting a constant intensity level;

8.

ROW—the image is levelled row by row. With the preselected absolute value of the gradients, the local extremes of the gradients are identified (see Figure 6). Between two such extremes, the pixels’ intensities of the given section are levelled. Configuration values: selection level of the absolute local extremes of gradients;

9.

COLUMN—analogous to block 8, only the columns are involved. With the preselected absolute value of the gradients, the local extremes of the gradients are identified (see Figure 6). Between two such extremes, the pixels’ intensities of the given section are levelled. Configuration values: selection level of the absolute local extremes of gradients;

10.

Mean—the arithmetic mean of two images (levelled in the direction of rows and columns in blocks 8 and 9) is calculated. The nature of the data in both directions can sometimes differ significantly. An important role here is played by the sharpness of the edges and the intensity distance between the individual levels (contrast between the shapes). Overall, with sufficient sharpness and contrast, there is no damage to the contours of the shapes, and the gradual application of the technical filter results in the levelling of the inside of the shapes;

11.

NLM filter—the result from the previous step is jagged and therefore needs to be smoothed. The filter parameters may or may not be the same as the parameters from block 2. Configuration values: DOS, SWS, CWS.

In blocks 4 and 6, it is necessary to generate a histogram for thresholding purposes.

Table 7. Histogram generation methods.

	Matlab Functions	Running Time
1	fitdist + pdf	1.5 [s]
2	histcounts + conv	4 [ms]

shows a comparison of two methods that could be used (the running times are given for block 4 where the input is a filtered microscopic image of 960 × 1280).

The first option was relatively time-consuming, with an average duration of around 1.5 s. Its output was a probability density function (PDF), while among the basic setting parameters that could be set were the kernel type and smoothing parameter. The more the nature of the real histogram differed from the Gaussian, the sooner significant distortions occurred when it was generated in this way, otherwise the selected Gaussian type helped to remove minor noise in the histogram.

The second option was significantly faster, with an average duration of around 4 milliseconds. It represents the splitting of the occurrence of intensities into a predetermined number of bins. In the case of a small number of bins, partial noise filtering occurred automatically on the histogram, but then there was a risk of losing important details. With a sufficient number of bins, e.g., 1000, the course of the histogram could be filtered with a suitable MA filter. Such a filter could be relatively fine and used sequentially several times (twice)—see Figure 8.

3. Results and Analysis

This section is divided into four subsections. Each of them demonstrates separate details:

• 3.1: Procedure A from Figure 7—the results for 12 randomly selected images were obtained with the preset configuration values depicted in Figure 7. There was no manual tuning of configuration values and the analysis was quick;
• 3.2: Procedure B from Figure 7—one particular image was chosen from Section 3.1 (the image of 1% Sn and Zoom4000). The simple application of the technical filter improved the accuracy from 92% to 94%. Again, there was no manual tuning of configuration values. The duration of the image analysis was longer than the time required by the application of the NLM filter (part of the technical filter: block 11, Figure 7);
• 3.3: Demanding image (tuning of the configuration values)—a demanding image of 1% Sn and Zoom2000, outside of the selection from Section 3.1, was chosen to demonstrate the complexity of the visual content. Multiple backgrounds were presented. Now, the configuration values (inclusive of the technical filter) were manually tuned. The accuracy had been improved from 79% to 92%.
• 3.4: Comparison of results with the XRD data—a set of 180 images were analysed. Most of them were processed with the preset configuration values. Minority of them had to be manually tuned. The value of the accuracy improvement could reach up to 20%. The XRD data were used just as a benchmark. They were not needed in actual analysis.

3.1. Procedure A from Figure 7

Table 8. Typical results of the classification and phase quantification.

Sn	Zoom4000		Zoom2000		Zoom1000
[%]	PQ [%]	Accuracy [%]	PQ [%]	Accuracy [%]	PQ [%]	Accuracy [%]
0	BG: 67.57 BW: 4.67 L: 27.76	99	BG: 65.43 BW: 4.38 L: 30.19	98	BG: 66.89 BW: 4.05 L: 29.06	98
1	BG: 71.50 BW: 6.05 L: 14.44 D: 8.01	92	BG: 69.21 BW: 3.59 L: 16.82 D: 10.38	85	BG: 73.54 BW: 3.32 L: 15.95 D: 7.19	91
2	BG: 76.76 BW: 5.40 L: 9.59 D: 8.25	95	BG: 71.00 BW: 3.27 L: 17.51 D: 8.22	95	BG: 70.94 BW: 3.25 L: 13.59 D: 12.22	82
3	BG: 73.66 BW: 3.87 L: 9.99 D: 12.48	95	BG: 80.03 BW: 4.73 L: 4.63 D: 10.61	93	BG: 79.63 BW: 5.50 L: 4.30 D: 10.57	95

shows the typical results that can be achieved by procedure A from Figure 7 (without using the technical filter). All Sn percentages are present: 0, 1, 2, 3, as well as all zoom levels: 4000, 2000, 1000. The accuracy of the classification was determined by qualified estimation and was defined as the percentage of correctly assigned pixels together for all classes against the number of all pixels in the image. Moreover, the image masks of individual classes obtained from the algorithm represented the phase quantification—a detail from a segmented and classified image can be seen on Figure 9.

Typical challenges in classification were present between classes BG and LD (presence of multiple backgrounds) and between classes L and D (very irregular histogram in block 6 of Figure 7).

Figure 9 depicts a masked detail from a Zoom4000 3% Sn image on the right side; the original image is on the left. The colour white stands for the BG background, red for the BW shapes, green for the L shapes, and finally, blue for the D shapes. Minor errors are obvious, especially at the edges of the D shapes and when distinguishing the BW and D shapes. The NLM filter reliably removed the emery paper scratches and precipitates, while sensitively preserving the important edges of shapes.

3.2. Procedure B from Figure 7 (the Technical Filter)

The technical filter could be applied to the image from Table 8 (1% Sn, Zoom4000). Figure 10 shows a part of the BG/LD(Shapes) histograms before and after the application of the technical filter (the bin indices between 650 and 900 from the total number 1000). The stimulus for the application of technical filter is the rel. high valley between two peaks marked as the point A. The technical filter transformed the data of the input image (red line) into a new image (blue line) whose histogram’s peaks are now more detached from each other, and the position of the appropriate valley (point B) is lower. The main aim of this transformation is to deepen the histogram. The courses of the histograms are intentionally unfiltered in this case.

The result of the technical filter application meant the inclusion of about 38,000 pixels in other classification classes. Besides minor unwanted effects on the contours of some shapes, there was a positive removal of errors, especially between the classes BG and LD(Shapes) and between L and D as well. The classification accuracy thus increased from the original value of 92% to 94%.

3.3. Demanding Image (Tuning of the Configuration Values)

As an example of more demanding data, the denoised microscopic image of 1% Sn at Zoom2000 shown in Figure 11 was chosen. The rows R38 and R904 are marked in red; the columns C538 and C709 are similarly marked in blue.

The set of four mutually related figures—Figure 12a of R38, Figure 12b of R904, Figure 13a of C538 and Figure 13b of C709—thus shows the relationships from this one microscopic image, Figure 11. Figure 12a,b represent the complete rows of the respective indices. Similarly, Figure 13a,b represent the courses of the complete columns of the respective indices.

In Figure 12a,b, the denoised NLM filter signal is presented in blue (Figure 7, block 2); the one-time application of the technical filter after averaging and filtering is presented in red (Figure 7, block 11) and the location of the necessary threshold levels (global separation of the BG background from the L + D shapes) is shown in green. Perpendicular courses of both columns C538 and C709 are indicatively marked in the form of arrows (relation to the horizontal axis). The areas A, B, C, D and E indicated in the upper part were characterised by their own intensity levels of background—multiple backgrounds due to channelling contrast. The number of such areas and their shape could not be predicted in advance. In this microscopic image, they had a longitudinal character agreeing with the direction of the columns. While in the case of R38 the threshold is at the level of 0.725, in the case of R904 it is already at the level of 0.775. The given difference is 0.775–0.725 = 0.05 ≈ 13 [pixels]. With the global thresholding, the same intensity value of the points in the entire image is assumed, which in this case is violated to such an extent that it exceeds the limits of the typical contrast differences between the BG and L shapes (see Table 5).

In Figure 13a,b, the denoised NLM filter signal is presented in blue (Figure 7, block 2), the technical filter levelling of the columns only is illustrated by magenta (Figure 7, block 9), the arithmetic mean of the technical filter levelling of the rows and columns is black (Figure 7, block 10) and the final technical filter output is red (Figure 7, block 11). Perpendicular courses of both rows R38 and R904 are indicatively marked in the form of arrows (relation to the horizontal axis). The drift of the intensity level is evident from both the blue and red lines, although it appears to be basically constant while observed with the naked eye in the microscopic image (Figure 11). These drifts may not be the same across all BG shapes. From Figure 13b, a similar character is evident with the shift in the threshold level described for rows R38 and R904. The accuracy of the data classification from Figure 11 in accordance with the procedure A from Figure 7 was firstly evaluated at the level of 79% with the preset configuration values. After some manual tuning of the configuration values and with the application of the technical filter, the accuracy reached 92%.

3.4. Comparison of Results with the XRD Data

The PQ results could be verified by the quantitative XRD data. It must be noted that the XRD measurements gave an average phase composition for a sample area of up to 100 mm². A single SEM image at Zoom1000 represents only 0.01 mm². However, this could be partially mitigated by choosing several measurement sites (images) along a sample. As could be seen from

Table 9. Comparison of results with XRD.

Sn [%]	Images	PQ ± Std.Dev [%]	XRD ± Std.Dev [%]
0	45	BG: 66.01 ± 2.01	BG: 66.30 ± 1.50
		BW: 3.37 ± 0.95	BW: 5.50 ± 0.40
		L: 30.62 ± 2.23	L: 28.30 ± 0.50
		Accuracy: 98.12 ± 0.74
1	45	BG: 72.31 ± 3.78	BG: 73.10 ± 1.00
		BW: 3.16 ± 0.82	BW: 4.90 ± 0.10
		L: 19.19 ± 3.52	L: 19.20 ± 0.20
		D: 5.34 ± 1.40	D: 2.80 ± 0.30
		Accuracy: 95.43 ± 3.64
2	45	BG: 75.58 ± 4.17	BG: 78.90 ± 1.80
		BW: 3.29 ± 0.69	BW: 5.00 ± 0.10
		L: 12.93 ± 4.60	L: 11.80 ± 0.60
		D: 8.21 ± 1.50	D: 4.30 ± 0.50
		Accuracy: 95.22 ± 3.39
3	45	BG: 78.93 ± 2.63	BG: 83.60 ± 2.00
		BW: 5.20 ± 1.08	BW: 4.20 ± 0.50
		L: 7.05 ± 1.65	L: 5.30 ± 1.10
		D: 8.82 ± 1.64	D: 6.90 ± 0.80
		Accuracy: 95.56 ± 1.94

, a good overlap was found between these different approaches of PQ.

For each sample (% Sn), 45 images were processed (always 15 images for each zoom level, 180 images in total). In some cases, better accuracy was achieved compared to Table 8 by the manual tuning of the configuration values. The column ‘PQ ± Std.Dev [%]’ indicates the results obtained by the methodology described for Table 8 without using the technical filter (Figure 7, procedure A). The column ‘XRD ± Std.Dev [%]’ gives the results from XRD, which are corrected as follows. The XRD analysis separately detected both components of the BW eutectoid ((η(Zn) + α(Al))), while IP resolved the eutectoid as one integral particle (BW shape). Therefore, with the simplified assumption that both eutectoid components were equally represented in the given particle, the range of α(Al) was multiplied by two, and the η(Zn) component (BG phase) was actually reduced by an additional α(Al) share. As indicated earlier, for the sake of simplicity, the proportion of aluminium (α(Al) phase) in the precipitates was not considered.

4. Discussion

Noise filter—In addition to the NLM filter (see Section 2, Noise and technical filter), the other options had been considered for noise removal. From the Matlab menu wiener2, medfilt2, modefilt and imboxfilt blurred the edges of shapes, as did imbilatfilt when entering larger sigma values. The couple imdiffuseest + imdiffusefilt, when used multiple times in a row, provided relatively nice results, but still there was a slight blurring of the edges. On the contrary, several precipitates were still maintained. Two options, localcontrast and locallapfilt, referred to as edge-aware processing [28], provided a pleasant contrast improvement for the human eye, but in fact they violated the condition of the same intensity value during further processing, e.g., for the histogram [43]. After verification, a finer NLM filter had been chosen with multiple sequential uses. Thus, the task of homogenising the segments in the form of removing noise and almost all precipitates was fulfilled, and it was possible to proceed to the next processing step. The segmentation logic for the inclusion or exclusion of the precipitates must be clarified in advance. The isolated occurrences of the precipitates within the background could be correctly reclassified in block 5, Corrections (see Figure 7).

Segmentation and classification—For the BW shapes, the level thresholding with the morphological operations had been chosen. Subsequently, for the BG, L and D segments, the histogram thresholding was used. In the case of morphology, sufficiently accurate results had been achieved. In this way, texture analysis [44,45] for the BW shapes, which took on too diverse forms, could be avoided. The histogram thresholding played a central role in segmentation and classification, and at the same time was a stumbling block.

The histograms can be used in a segmentation process, provided that the objects can be distinguished from each other based on their intensity values [46]. In general, the threshold is placed in a distinct deep valley in the histogram [47]. However, if such a valley does not exist, the placement of the threshold is difficult (especially in the case of algorithmisation for a computer).

Most real images are not characterised by ideal histograms; on the contrary, several challenges can appear [48,49,50]:

• Flat and wide valleys;
• Two peaks characterised by significantly different heights (a problematically detectable valley);
• Noise in the histogram;
• Noisy spikes;
• Overlapping distributions of BG and the object;
• The object represented in a small percentage;
• BG consisting of several regions.

There are different approaches to deal with the challenges of noise and modalities in histograms: analysis using convex hulls [49], wavelet transformation [51], or smoothing using a kernel [52]. The division of the threshold techniques is not uniform; more information can be found in [51,53,54,55].

Regardless of manual or automatic processing, these can clearly be included among the advantages of histogram thresholding:

• All segmentation areas are classified at the same time;
• Everything on one side from the threshold level automatically falls into one segment/class, and thus the problem with oversegmentation is eliminated (see the white rim described in Figure 3);
• Adaptability to changes in brightness (horizontal shift of the histogram);
• The same sense of all intensity values in the image (global thresholding), which, however, can easily turn into a disadvantage if the histogram deployment conditions are not observed.

The disadvantages of histogram thresholding are as follows. Pure intensity decision-making about segmentation and classification has a disadvantage in the case of differences in the meaning of the individual values that are represented in an image. There were some occasional cases where the D shapes were formed by the particles with a high-intensity interior. Their interior parts were classified incorrectly by the given histogram thresholding. An illustrative example of the situation is shown in Figure 14 (a detail of a Zoom2000 image from the 1% Sn sample).

When generating a histogram, in addition to the two mentioned options in Table 7, Matlab also offered imhist, which was closer in nature to the second option from Table 7 (histcounts) and histfit, which internally used the function from the first option of Table 7 (fitdist). For these reasons, they had not been described. The function fitgmdist represents a Gaussian mixture model. The need to enter the number of components (modes) in advance had been considered as the main disadvantage. In this way, the shape of the histogram could be distorted.

Table 10. Factors affecting the BG/LD(Shapes) and L/D histograms.

	Factor	Manifestation	Attribute	Operator
1	material drift	segment horizontality	objective
2	grain orientation (channelling)	multiple backgrounds	objective
3	spatial representation	number, form and intensity of the shapes	objective
4	Sn content	contrast differences	objective	informed
5	application of filters	noise removal, segment horizontality		determines
6	microscopy parameters	noise character, brightness, contrast		determines

presents the factors affecting the shape of histograms, and subsequently the accuracy of segmentation and classification based on the histogram thresholding. The input represents a filtered microscopic image or an image that is a product of the technical filtering. Some factors or manifestations can be separated from each other, and some are more or less interconnected.

Table 11. Characteristics of the BG/LD histograms (block 4, Figure 7) for the individual zoom levels.

Zoom	Typical Characteristics of a Histogram	Human Readability of Threshold
1000	practically unimodal	problematic
2000	1, 2 or 3 modes, often non-Gaussian
4000	mainly 2 modes, sometimes 3 modes	easy

lists the typical characteristics of the BG/LD(Shapes) histograms (see block 4 from Figure 7) for the individual zoom levels. The importance of the zoom aspect must be emphasised, because it directly affects the spatial representation of the shapes (third factor, Table 10) and with a numerically smaller zoom, there is a greater probability of encountering several areas with different grain orientations e.g., channelling contrast (second factor, Table 10).

In the rare case of demanding images with a complex visual content, the shape and character of the L/D histogram (block 6, Figure 7) showed much greater variations in the number of modes and their shape and character, compared to the BG/LD(Shapes) histogram. For this reason, setting the threshold level by the operator was necessary.

Technical filter—The noise filter reliably removed noise and faithfully provided the supposed ground-truth information (the real state of the ground-truth was unknown). On the contrary, the technical filter, in addition to removing the possible fluctuations, also adjusted the image itself in order to facilitate further processing [56] (the ground-truth status is true, but not quite ‘satisfactory’ for the next step). The deployment of the technical filter was expressly at the discretion of the operator. Its essential philosophy was not to harm the edges of shapes and gradually level the individual sections. This all served explicitly for histogram editing purposes (deepening), which improved the segmentation and classification results. After that, its output in the form of a transformed image was no longer needed.

The application of the technical filter also had its disadvantages:

• Small erosion of less sharp shape edges (especially with multiple uses of the filter);
• Segmentation line artefacts occurring if the gradient level was incorrectly determined;
• Different effects at the different zoom levels and Sn content;
• No recommendation for the number of repetitions.

Images—The microscope also offered the option of saving the images with a 16-bit grayscale depth. However, upon inspection, it was found that this was only a formal multiplication of the 8-bit depth (amplitude), and therefore, no new information content was available. An image could be saved in different formats. Russ [15] does not recommend using .jpeg. It reduces the information content and causes the creation of unwanted effects. Any lossy compression should be avoided. The recommended format is .tiff.

Cooperation of experts—Chen [57] underlines the teamwork. The details of the task (e.g., membership of the precipitates to the relevant class) and the verification of phenomena in the displayed data (material drift presented in Figure 13a,b) demanded a cooperation between the technological expert and IP expert.

Difficulty—The difficulty of microstructure segmentation is given by the following visual characteristics [20]:

• High resolution;
• Extreme variability in texture and shapes;
• Significantly fragmented images with often unclear boundaries of displayed objects;
• Unbalanced classes (unequal percentage representation on the image);
• Absence of prior structural information.

Despite being professionally acquired, the images of microstructures showed small variations in brightness. This aspect was demonstrated as a horizontal shift of histograms (see Section 4, Discussion—advantages of histogram thresholding). Moreover, the complex visual content provided exhibited significant variations of scenes, shapes, textures and multiple backgrounds. Two zoom levels (1000, 2000) and two Sn contents (1%, 2%) were the most complex, and therefore demanding.

Challenges—The difficulty of visual characteristics of microstructures poses the challenges that need to be addressed [20]:

• Acquisition and sharing of detailed datasets within the scientific community [58];
• The need to speed up preannotation (significantly facilitating the work of experts);
• Deployment of pretrained DL models;
• Providing open-source code for scientific purposes [59].

The presented approach can be used for other data with similar visual characteristics in order to easily prepare the training dataset in the scientific community. Then, some manual refinement is needed (approx. from 1 to 5%).

5. Conclusions

The SEM BSE imaging modality was chosen intentionally for the given type of task. The results of the presented approach were confirmed with another two independent modalities: EDX and XRD. Despite considerable effort, it was not possible to fully automate the given approach. The segmentation and classification algorithm contains a number of configuration values. These had been preset based on the expertise from the algorithm development. For most cases, they were sufficient and quickly provided high-quality results. In rare cases of images with a complex visual content, they could be manually tuned in order to improve the results.

For these reasons, the approach is declared as semiautomatic when the input of the human factor is required in the individual steps, either for assessing the achieved results, entering the histogram thresholds or for setting the configuration values in the algorithm. The main cause is apparently the orientation only on the meaning of intensity values, which in more complex situations violates the conditions of applying the global histogram thresholding.

The results of noise filtering proved to be fully satisfactory, and in the case of mastering high-quality segmentation and classification, they could be directly used for further analysis of the individual classes.

Overall, the results of the research can be directly processed in the field of materials science or serve as an intermediate step for the preparation of training data for deep learning, and thus save energy and time. Here, the identification of the most demanding images is very useful, because it is exactly these ones that will be of great importance for deep learning.