1. Introduction
In the industrial manufacturing and biomedical industries, there are many situations in which three-dimensional (3D) spatial measurement, tracking and identification of the targets of interest in a volume are required. Among these scenarios, one or all of status information including the dimensions, the locations and the materials are required to be detected due to the working conditions, operation flow, process control, and other requirements. According to the numerous different applications in consumer electronics, car industries, plane industries, or biomedical research and industries, transparent objects must meet different quality requirements with compound measurements of the target of interests. Usually, visual observation or automated machine vision techniques can partially fulfill the abovementioned tasks on the premise of human intervention or complicated and inefficient measurement processes, which obviously cannot meet the actual production requirements in terms of multiple measurands, measuring speed and on-machine measurement.
With the continuous advancement of science and technology, various methodologies and techniques have been developed to perform measurements of various measurands, including dimensions [
1], spatial locations [
2,
3] and materials [
4,
5,
6]. Three-dimensional measurement methods, such as light scattering techniques [
7], fiber interferometry [
8] and optical slope sensors [
9], usually measure the surface profile rather than targets of interests in a volume or a free 3D space. Stereovision is an effective way to detect 3D objects, but often requires multiple image sensors, complex calibration process and high requirements of rigidity to keep the baseline. Fringe projection profilometry is another potential method to measure various targets in a volume, but the projected pattern has high vulnerability when facing semi-transparent medium and targets in motion. Moreover, considering the measurements in transparent material or in a volume, optical coherent tomography (OCT) can be used in glass inspection [
10]. OCT can obtain clear images and defect depth, but the inevitable scanning makes its detection speed greatly reduced.
Autostereoscopic metrology is an emerging technology that was originally developed for the on-machine measurement of micro-structured surfaces [
11]. The compact system design can provide fast data acquisition of 3D raw data through the incorporation of a micro lens array (MLA) and high accuracy in three-dimensional (3D) reconstruction due to the reversibility of optical rays in the recording process and reconstruction process of this autostereoscopic measurement method. The autostereoscopic 3D metrology method has very strong advantages in measuring targets in free 3D space, such as in a volume or a transparent material, since it has good extendibility of measuring scale from the macroscopic to the microscopic level and has few limitations of the optical information receiving process since the image sensors can receive reflected or refracted light signals from different targets and medium.
While measuring the three-dimensional information or locating certain targets in a 3D space using 3D measurement technologies, identifying the material is another measurement task of great importance. Spectrometry is an effective way to analyze the composition of substances. Among various spectral analysis techniques, Raman spectroscopy is an important technical means to quickly and non-destructively characterize the structure and composition of materials [
12]. Raman spectroscopy has a wide range of applications in the field of materials research. Using Raman spectroscopy to monitor average degree of smoothness and extent of structural defects during materials synthesis to study the evolution of growth kinetics to optimize growth conditions [
13]. Raman spectroscopy also has great potential in microbial detection. The use of Raman spectroscopy can rapidly identify causative agent and its susceptibility to antimicrobials [
14], and the combination of Raman tweezers and microfluidics enables real-time analysis and identification of single microbial cells [
15]. Rapid microbial detection can shorten hospital stay, reduce mortality and reduce costs of medical. It is likely to become a reliable and fast diagnostic tool in the future. Confocal Raman spectroscopy combines confocal microcopy and Raman spectroscopy, and is widely used in the fields of physical chemistry and biomedicine for its non-destructive testing, tomography, and molecular fingerprint characteristics [
16,
17,
18]. High-lateral resolution and well-resolved depth profiling with the confocal Raman spectroscopy allows it to identify tiny targets and investigate hidden phases underneath the surface without destruction of extraterrestrial materials [
19]. By adopting the reflective illumination light path, the designed confocal micro-Raman system has the enhanced intensity of the Raman spectrum signal and the improved signal-to-noise ratio [
20]. In the field of in vivo confocal Raman spectroscopy, detection methods can be divided into the following two types. For the first type of system, confocal Raman spectra were obtained without imaging guidance. During the measurement, the biological tissue of interest is pressed on the cover glass, and Raman spectra of different depths are obtained by vertical scanning [
21]. The second type of system combines Raman microscopy with reflectance confocal microscopy (RCM), in which Raman microscopy and RCM work alternately. RCM imaging is used to identify points of interest (POIs). The RCM imaging channel was then closed to acquire micro-Raman from a predetermined POI [
22]. In order to expand application scenarios, methods to improve resolution have been proposed. Combining the confocal Raman spectroscopy imaging with radially polarized light compact focusing, the spatial resolution of confocal Raman spectroscopy imaging can be improved by compressing the incident spot diameter and image restoration techniques [
23]. As a new type of Raman spectroscopy detection method, spatially offset Raman spectroscopy (SORS) can effectively suppress the interference of surface component Raman spectroscopy and fluorescence background. Combined with appropriate data processing methods, it is especially suitable for translucent or deep spectral information extraction of substances with highly fluorescent background materials [
24]. Its detection depth is usually from 20 μm to 5 mm, which exceeds the detection depth limit of confocal Raman spectroscopy [
25]. Samples from which spectra were collected do not need to be prepared and can be measured under high-temperature, low-temperature and high-pressure environments. The instrument is stable, moderate in size, has a low maintenance cost and is simple to use [
26]. However, locating the right excitation position in order to precisely identify the material has always been a challenge in the abovementioned spectroscopic methodologies.
When using the autostereoscopic 3D measurement method to measure the spatial position of the target, the robustness of judging the depth by the sharpness function of the focus position is poor. There are two main reasons for this shortcoming. One is the unapparent visual characteristics of the object hinder the judgment of the algorithm, and the other one is that the optical system has a depth of field, and the ability to distinguish different depths of focus is not strong. Furthermore, it is not possible to discriminate the substance composition only using a visual inspection system. Using Raman spectroscopy to discriminate material composition is a good choice. However, when the existing spectroscopic system, such as confocal Raman spectroscopy, detects sparsely distributed targets in a three-dimensional space, the imaging system needs to scan the entire depth to find the target because the microscopic imaging system has a small depth of field. This scanning process obviously consumes a lot of time, and may cause system drift. In addition, SORS cannot accurately obtain the spatial position of the target. Based on the analysis above, there is currently no compound measurement method that can simultaneously perform spatial localization and 3D measurement in a non-scanning manner as well as the identification in a volume. In this paper, we present an Autostereoscopic-Raman Spectrometry-based (ARS) three-dimensional compound metrology system for measurements, tracking and identification in a volume. The ARS method avoids the mapping process and detects much faster than confocal Raman spectroscopy, which give it great potential for industrial applications.
2. Autostereoscopy-Raman Spectrometry-Based Measurement Methodology
The autostereoscopic 3D metrology method [
27] makes use of a micro lens array (MLA) that is incorporated into a single-lens imaging system to capture the raw 3D information of the measured surface in a single snapshot. As a machine vision-based measurement method which uses pixel information as the main medium, the diversity of pixel information distribution (PID) determines the resolution in regard to the matching process. Here, the diversity of PID can be interpreted as certain regional pixel information has pixel values within a large range.
In the recording process, the principle of autostereoscopic 3D metrology method system, they use EIs that contain multi-viewpoint stereo information to record spatial targets. As shown in
Figure 1, the virtual MLA simulated in computer and a 3D reconstruction algorithm are applied to calculate and rebuild the target in object space. By applying this special digital process, the target 3D information can be obtained quickly and easily, and it is more convenient to handle following measurement data [
27,
28]. The 3D target reconstructed by computed depth tomography algorithm (CDTA) is overlapped by a series of sharp depth slices. This process consists of three steps. First, the depth slices are obtained by re-mapping of the pixels from each EI. Second, the clear parts of depth slices are extracted by calculating sharpness function of each pixel. Finally, the clear parts are spliced to realize the reconstruction of the 3D target.
Specifically, to obtain image sequence of depth slices is a process of re-mapping the pixels of EIs by virtual MLA. It is an inverse process of recording. It projects each 2D EI onto a special depth plane in object space through optical axis of its respective elemental lens. In the image of a certain depth slice, only the object points belonging to the corresponding depth are clear, while the foreground and background are blurred, because this certain depth is finely focused and other depth planes are defocused. Based on the description of the recording process, the disparity can only be determined by the depth information in the object space. In other words, the disparity of the same depth plane is invariable, and the disparity of different depth planes is variable.
In the reconstructed image sequence of depth slices, different points have different focus levels and show as discs of confusion or focus points on a certain slice. The optimum focus position of a point in depth can be determined by calculating the maximum focus level of each point.
The sharpness function
is used to evaluate the focus level of a point, and it depends on the gray difference contained in the point and its small local neighborhood, in which the size is generally
,
and
. As shown in
Figure 1, using the
neighborhood for example,
,
is the total number of image sequence,
is the gray value of pixel
in
th slice,
, and
means the region of EI. The center distance of consecutive pixels in the horizontal and vertical directions is normalized to 1, and the center distance of diagonal direction is normalized to
.
Figure 1 shows the sharpness evaluation curve
of each image point
in the EI region
is generated by the vector,
The optimum focus position
of image point
can be characterized by the number
of image sequence,
in region can form a 2D matrix by calculating each image point. Finally, the height of the target can be reconstructed by mapping the focusing position to actual depth information.
Compared with other reconstruction methods, the CDTA does not require a complicated image matching process, but only perform re-projection of EIs by MLA. It avoids the matching error of corresponding image points that greatly impact reconstruction accuracy. At the same time, CDTA is free to select a specific depth region to reconstruct 3D information of object space, even only segmentation of the target to meet the inspection requirement.
However, this CDTA method has encountered an unavoidable practical problem especially when facing with an object with limited features but with a larger thickness. It usually brought challenges to precisely identify the right peak of sharpness among multiple peaks. Considering the low robustness of the of current focus measure function (FMF), additional information needs to be incorporated to jointly identify the peak of the sharpness of certain object points in a series of the reconstructed image sequence of depth slices.
Raman spectroscopy is an important technical means to quickly and non-destructively characterize the structure and composition of materials. The sample does not need to be prepared and can be measured under various external environments with high tolerance. Raman spectral signal is a unique and effective signal to differentiate and identify different targets in a volume, especially when the target and medium in the volume are different [
29]. In this sense, additional information is needed to provide solid and robust features on sharpness detection of certain object points among multiple interference peaks in a series of the reconstructed image sequence of depth slices. However, high requirements for the excitation and collection are required by the Raman spectroscopy, which means it is necessary to accurately locate the excitation position in order to effectively collect the spectral signal. Usually, this requires additional repeated operations, including moving to multiple trial excitation positions, outputting Raman spectral signal, comparing these signals and confirming one position as the right excitation position.
In this sense, in order to solve the problem of evaluating the fine focused point in a series of the reconstructed image sequence of depth slices generated by the autostereoscopic measurement method, to solve the problem of locating the exact excitation position of Raman spectroscopy, and to trace, measure and identify the targets of interest in a volume, we propose this comprehensive Autostereoscopic-Raman Spectrometry-based (ARS) three-dimensional measurement method, showed in
Figure 2. As a compound measurement method, the sharpness function signal of autostereoscopic method and Raman spectral signal are providing cross-validation to each other.
The distribution of sharpness of a certain object point in a series of the reconstructed image sequence of depth slices generated from autostereoscopic method is able to guide spectral detection for the excitation position in order to receive the most significant and accurate spectral distribution curve for material identification. Meanwhile, the intensity of spectral detection can in return guide the precise detection of the true peak of the sharpness.
Accordingly, this compound signal of ARS 3D measurement method is able to simultaneously locate the exact spatial position of a certain point in a volume, measure the spatial dimensional information and identify the material of targets of interest in a volume.
4. Experimental Studies
In order to verify the proposed Autostereoscopy-Raman Spectrometry-based measurement methodology and its associated measurement system, which is shown in
Figure 5. As for the sample preparation for the verification of the proposed method, we specifically construct a multi-layer-structure and put three different kinds of objects between the cover glass and the substrate glass slide as test samples, including non-woven fibric from shoe covers, non-woven fabric from masks, meltblown fabric and anti-static clothing fibers, as show in
Figure 6.
Figure 6a is the first sample, which contains anti-static clothing fibers, non-woven fibric from shoe covers, and non-woven fibric from masks. The depths of the three targets are 40 microns, 80 microns and 120 microns, respectively.
Figure 6b is the second sample, we replaced the fiber of the second layer with meltblown fabric and changed the depth of the target. The new depths were 50 microns, 80 microns and 110 microns. The meltblown fabric very different from other fibers, as vividly shown in
Figure 6b, which can further verify the robustness of our method. The design of the target sample is to simulate the pollutants attached between glass interlayers which is especially similar to the industrial manufactured sample.
We simulated the contaminants in different thicknesses of glass interlayers by artificially locating them at different depths away from focal plane of the objective in order to provide the true value of depth, which can be used as reference data of experimental studies. To imitate the structure of the first sample, we artificially located the three targets at 40 microns, 80 microns and 120 microns away from the focal plane by moving the z axis, which has repeatability of 0.1 microns. For the second sample, the related offsets are 50 microns, 80 microns and 110 microns, respectively.
Figure 7a,d,g are 2D images of anti-static clothing fibers, non-woven fibers from masks and non-woven fibers from shoe covers captured by a Nikon DS-Fi3 microscope camera under a 20× microscope. Since some of the CCD pixels are invalid, we do a basic cropping of the image.
Figure 7b,e,h are the elemental images of the above-mentioned fibers, and
Figure 7c,f,i are the reconstructed images.
Figure 8 shows the 2D images, EIs and reconstructed images of the second sample. By processing the data via the purposely developed reconstructed algorithm and image evaluation function, we will get the sharpness evaluation function graph as shown in
Figure 2 to roughly locate the spatial position of the target. Usually, there will be multiple peaks regarding different depths. We traverse these peaks and collect the associated Raman spectrum. As shown in
Figure 2, Raman spectrometry signals with high signal-to-noise ratio are often obtained at the noise peaks, but they are mistaken for focal depth. Raman spectrometry signal can provide additional information to rule out the false peaks as noise. Then, the proposed ARS system is able to get both accurate location information and material composition information simultaneously.
As shown in
Figure 9, the spatial positions of the six different types of fibers from the two samples are reconstructed by the autostereoscopic module of the proposed ARS system and the sharpness function from 0 μm to 150 μm are plotted accordingly. It can be seen that due to the calibration error of the autostereoscopic module, the calculated position is different from the actual position. The robustness of the main peak as a condition to determine the depth position is not high, and there are some other peaks around the main peak, and the peak of the correct spatial positions may be located among them. In this sense, Raman spectra at these positions, which are recognized as the peaks of the sharpness function, are collected by the Raman spectrometry module to verify and calibrate the calculated spatial positions. In this experimental study, all peaks within ±15 µm from the main peak are collected with the corresponding Raman spectral signal. Among these positions, the one with the highest signal-to-noise ratio of the Raman spectrum can be determined as the confirmed spatial position. The time resolution of the proposed ARS system mainly depends on two aspects. Autostereoscopy measurement methodology can finish the data acquisition process through single snapshot and can calculate all image stacks within 1 s, as well as initially identify the target depth range simultaneously. On the other hand, the acquisition of the Raman spectrum can be completed in a few hundred milliseconds to a few seconds. In this sense, the time resolution of the proposed ARS system is within 1 s. To be noticed, the calculated depth positions cross-checked and confirmed by Raman spectral correction of three different types of fibers are compared with the true depth, which is obtained by the working distance and defocusing amount, the average error of 15 repeated measurement process is within 1 μm. The spectrum collection success rate of this equipment is 98%.
Figure 9 also shows the Raman spectra of the three types of fibers, the Savizkg-Golay filter is used for smoothing and the adaptive iteratively reweighted Penalized Least Squares algorithm proposed by Zhang [
32] is used to remove the baseline. Materials from different sources have distinct characteristic peaks that can be clearly distinguished. By building a database of Raman spectra of possible contaminants, we can easily classify contaminants in practical detection.
5. Discussion & Conclusions
In this paper, an Autostereoscopic-Raman Spectrometry-based (ARS) measurement methodology and its associated measurement system is proposed and developed accordingly, which can perform 3D positioning, 3D measurement and identification of material composition structure of certain depth at the same time in volume, that is verified by the experimental studies in this paper. Moreover, the research of the proposed ARS system shows the potential abilities of detecting multilayer glass structures without scanning, accurately locate the spatial position of impurities, and conduct material analysis through the proposed ARS system. As a pilot study, the autostereoscopic module and Raman spectrum module are currently add-on modules for commercial microscopes without changing the original system structure. A 20× objective that fixedly mounted on this commercial microscope was used to verify the performance of the proposed ARS system by testing the 3D positioning and identification capabilities in a volume that is constructed by a multiple-layer glass structure and three types of fibers. After Raman spectral signal correction, the average positioning error of the ARS system was sub-micron level, and the success rate of spectrum collection has been enhanced to 98%.
The proposed ARS system has proved its high potentials for locating, tracing, measuring and identifying various sources of impurities to accurately control the industrial production process and improve the yield rate with low-cost and high-efficiency methods. It is interesting to note that this ARS system is able to have a larger field of view and depth of field which are obtained by quickly altering the objective magnification and matched micro lens array, enabling better measurement scale and efficiency for industrial scenarios. In this sense, this comprehensive measurement method will have a wider range of applications in the future.